[ab-advisor] A/B experiment: sub_agent_strategy for agent-persona-explorer#33753
Merged
pelikhan merged 3 commits intoMay 21, 2026
Merged
Conversation
Closed
8 tasks
…xplorer Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add experiment campaign for agent-persona-explorer with A/B test
[ab-advisor] A/B experiment: sub_agent_strategy for agent-persona-explorer
May 21, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Implements an A/B experiment (sub_agent_strategy) for agent-persona-explorer to compare per-scenario sub-agent invocations vs. a single batched sub-agent invocation, aiming to reduce token usage while maintaining quality. Also includes regenerated workflow lockfiles reflecting updated generated firewall/model alias configuration.
Changes:
- Adds an
experiments.sub_agent_strategyfrontmatter block toagent-persona-explorer.md(variants, metrics, guardrails, analysis metadata). - Adds Phase 3 conditional prompt logic to switch between batch and per_scenario sub-agent invocation strategies.
- Regenerates multiple workflow
*.lock.ymlfiles (notably updating the embedded AWF config model alias map).
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/agent-persona-explorer.md | Adds sub_agent_strategy experiment metadata and prompt conditionals for batch vs per-scenario execution. |
| .github/workflows/workflow-health-manager.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/workflow-generator.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/video-analyzer.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/test-workflow.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/test-quality-sentinel.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/test-project-url-default.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/test-dispatcher.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/super-linter.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/slide-deck-maintainer.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/security-review.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/security-compliance.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/repo-tree-map.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/refiner.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/q.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/pr-description-caveman.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/poem-bot.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/plan.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/pdf-summary.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/metrics-collector.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/mergefest.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/lint-monster.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/jsweep.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/gpclean.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/example-permissions-warning.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/dev.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/dev-hawk.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/craft.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/contribution-check.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/code-simplifier.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/brave.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/bot-detection.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/archie.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
| .github/workflows/ace-editor.lock.yml | Regenerated lockfile; embedded AWF config/model alias map updated. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 120/234 changed files
- Comments generated: 1
| description: "Test whether batch scenario testing reduces token costs vs. per-scenario sub-agent calls" | ||
| hypothesis: "H0: no change in effective_tokens or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss" | ||
| metric: effective_tokens | ||
| secondary_metrics: [run_duration_minutes, scenarios_tested, output_quality_score] |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the
sub_agent_strategyA/B experiment campaign foragent-persona-explorer, testing whether consolidating all scenario testing into a single sub-agent call (batch) reduces token costs vs. the current per-scenario invocation approach.Frontmatter changes
experiments: sub_agent_strategyblock with variants[per_scenario, batch], 50/50 weight,t_testanalysis,effective_tokensas primary metric, and guardrails requiringdiscussion_created == 1andscenarios_analyzed >= 3issue: #aw_campaignplaceholder omitted — parses asnullin YAML (comment marker), failing the schema'sintegerconstraint; can be added when a campaign issue number existsdirection:sub-field on guardrail items replaced with the schema-compliant combinedthreshold:form (e.g."==1",">=3")Phase 3 conditional blocks
{{else}}): original behavior preserved unchangedLock file
Regenerated via
gh aw compile --approve. The--approvewas needed because the prior lock referenceddocker/build-push-actionanddocker/setup-buildx-actionthat are no longer emitted — a pre-existing delta unrelated to this change.