Skip to content

Surface experiment metadata fields in picker step summary#34665

Merged
pelikhan merged 4 commits into
mainfrom
copilot/ab-advisor-improve-experiment-infrastructure
May 25, 2026
Merged

Surface experiment metadata fields in picker step summary#34665
pelikhan merged 4 commits into
mainfrom
copilot/ab-advisor-improve-experiment-infrastructure

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 25, 2026

analysis_type, tags, and notify were already compiled from workflow frontmatter but were effectively inert in the JS picker runtime. This change makes those fields operational in runtime reporting by surfacing them in the assignment summary emitted by pick_experiment.cjs.

  • Step summary now includes schema metadata per assignment

    • Added an Assignment Details section in pick_experiment.cjs for each selected experiment.
    • Emits a compact Field/Value table with:
      • Experiment
      • Assigned variant
      • Analysis type
      • Run count (this variant) (with min_samples target when present)
  • tags and notify are now runtime-visible

    • When configured, summary includes:
      • Tags row (formatted list)
      • Notify row (discussion/issue targets)
  • Focused coverage for the new behavior

    • Added a targeted test case in pick_experiment.test.cjs asserting summary rendering for:
      • analysis_type
      • tags
      • notify
      • min_samples-aware run count formatting
detailLines.push(`| Analysis type | \`${analysisType}\` |`);
if (tags.length > 0) {
  detailLines.push(`| Tags | ${tags.map(tag => `\`${tag}\``).join(", ")} |`);
}
if (notifyTargets.length > 0) {
  detailLines.push(`| Notify | ${notifyTargets.join("; ")} |`);
}

Copilot AI and others added 2 commits May 25, 2026 14:37
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…mmary

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix experiment infrastructure: complete frontmatter schema Surface experiment metadata fields in picker step summary May 25, 2026
Copilot AI requested a review from pelikhan May 25, 2026 14:40
@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot use progressive disclosure, details element and same style of headers as other reports

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan marked this pull request as ready for review May 25, 2026 14:48
Copilot AI review requested due to automatic review settings May 25, 2026 14:48
@pelikhan pelikhan merged commit 4271625 into main May 25, 2026
@pelikhan pelikhan deleted the copilot/ab-advisor-improve-experiment-infrastructure branch May 25, 2026 14:48
Copilot stopped work on behalf of pelikhan due to an error May 25, 2026 14:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes experiment frontmatter metadata (analysis_type, tags, notify) visible at runtime by emitting it into the GitHub Actions step summary produced by pick_experiment.cjs, and adds a focused test to validate the new summary rendering.

Changes:

  • Extend pick_experiment.cjs step summary with an “Assignment Details” section (per selected experiment) including analysis_type, tags, notify, and min_samples-aware run count formatting.
  • Add a targeted test case covering the new summary output.
  • Regenerate/update multiple workflow *.lock.yml files (firewall config/model allowlists embedded in workflows).
Show a summary per file
File Description
actions/setup/js/pick_experiment.cjs Adds per-assignment metadata rendering to the step summary output.
actions/setup/js/pick_experiment.test.cjs Adds a test asserting the new summary content for analysis_type, tags, notify, and min_samples formatting.
.github/workflows/workflow-health-manager.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/workflow-generator.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/video-analyzer.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/test-workflow.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/test-quality-sentinel.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/test-project-url-default.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/test-dispatcher.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/super-linter.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/slide-deck-maintainer.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/security-review.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/security-compliance.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/repo-tree-map.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/refiner.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/q.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/pr-description-caveman.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/poem-bot.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/plan.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/pdf-summary.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/metrics-collector.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/mergefest.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/lint-monster.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/jsweep.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/gpclean.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/dev-hawk.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/craft.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/contribution-check.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/code-simplifier.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/brave.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/archie.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/agentic-token-optimizer.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/agentic-token-audit.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.
.github/workflows/ace-editor.lock.yml Updates embedded firewall/model allowlist config in the locked workflow.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 116/237 changed files
  • Comments generated: 2

Comment on lines 232 to +236
const runsForExp = state.runs ? state.runs.filter(r => r.assignments && name in r.assignments) : null;
const totalCount = runsForExp !== null && runsForExp.length > 0 ? runsForExp.length : Object.values(/** @type {number[]} */ counts).reduce((a, b) => a + b, 0);
lines.push(`| \`${name}\` | **${selected}** | ${thisCount} / ${totalCount} |`);

const cfg = configs[name] || {};
Comment on lines +254 to +256
detailLines.push("| Field | Value |");
detailLines.push("| --- | --- |");
detailLines.push(`| Experiment | \`${name}\` |`);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ab-advisor] Improve experiment infrastructure: schema, reporting & audit

3 participants