Analysis: branch storage supports multiple experiments per workflow ID; simplification to single-experiment API not recommended

[Copilot]

Assessed the repo-branch experiment storage strategy and evaluated whether the API should be collapsed to one experiment per workflow ID.

Branch storage

The current repo storage mode uses a single branch per workflow (experiments/{sanitizedWorkflowID}) with a single state.json whose counts object is keyed by experiment name. Multiple experiments per workflow ID are fully supported today — no structural change needed.

{
  "counts": {
    "style":   { "concise": 5, "detailed": 4 },
    "caveman": { "yes": 3, "no": 3 }
  },
  "runs": [{ "run_id": "...", "assignments": { "style": "concise", "caveman": "yes" } }]
}

All experiments for one workflow share one branch, one state.json, and one push_experiments_state job — cost is fixed regardless of experiment count.

Single-experiment API simplification

Not recommended:

• No runtime simplification. The branch, state.json, pick_experiment.cjs, and push_experiments_state all operate identically whether there is 1 or N experiments. A schema-level restriction would add enforcement code without removing any runtime complexity.
• Legitimate multi-variate use cases exist. Teams testing orthogonal dimensions (e.g., prompt tone and response length independently) need the named-map form without splitting into separate workflow files.
• The real concern is documentation, not enforcement. Interaction effects from simultaneous experiments are a usage concern, not an API defect. Guidance ("run one experiment at a time unless your design accounts for interactions") is the right mitigation.

The one genuine awkwardness — the storage reserved-key collision inside the experiments map — could be addressed independently by promoting storage to a top-level key (e.g., experiments_storage: repo) without collapsing the multi-experiment capability.

[Copilot]

Pull request overview

Updates a set of GitHub Actions workflow lockfiles by changing the Copilot CLI --allow-tool shell(...) allowlist entries (and their generated comments), primarily by replacing previously specific commands containing single quotes with shorter “prefix” forms.

Changes:

• Regenerates Copilot CLI tool allowlists in multiple *.lock.yml workflows (notably find, grep, git log, jq patterns).
• Updates the corresponding “Copilot CLI tool arguments (sorted)” comment blocks to match the new allowlist entries.
• Normalizes/loosens some command allowlists by removing quoted patterns and paths from the shell(...) strings.

Show a summary per file

File	Description
.github/workflows/workflow-skill-extractor.lock.yml	Updates shell(find ...) / shell(grep ...) allowlist entries and their comment block.
.github/workflows/ubuntu-image-analyzer.lock.yml	Updates shell(find ...) allowlist entry and corresponding comment.
.github/workflows/spec-librarian.lock.yml	Truncates several find/grep/git log allowlist entries to prefix forms; updates comments accordingly.
.github/workflows/spec-extractor.lock.yml	Updates find/grep allowlist entries and comments.
.github/workflows/layout-spec-maintainer.lock.yml	Simplifies find/grep/yq allowlist entries; updates comments accordingly.
.github/workflows/discussion-task-miner.lock.yml	Updates find allowlist entry; updates comments accordingly.
.github/workflows/delight.lock.yml	Updates find allowlist entries for docs/workflows; updates comments accordingly.
.github/workflows/daily-testify-uber-super-expert.lock.yml	Truncates find/grep allowlist entries; updates comments accordingly.
.github/workflows/daily-safe-output-integrator.lock.yml	Truncates find/grep allowlist entries; updates comments accordingly.
.github/workflows/daily-mcp-concurrency-analysis.lock.yml	Truncates find/git log/grep/jq allowlist entries; updates comments accordingly.
.github/workflows/daily-file-diet.lock.yml	Truncates find/grep allowlist entries; updates comments accordingly.
.github/workflows/daily-compiler-quality.lock.yml	Truncates find/git log/grep allowlist entries; updates comments accordingly.
.github/workflows/copilot-cli-deep-research.lock.yml	Truncates find pkg -name ... allowlist entry; updates comments accordingly.
.github/workflows/ab-testing-advisor.lock.yml	Truncates several find/grep allowlist entries; updates comments accordingly.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 14/14 changed files
• Comments generated: 2