Optimize ab-testing-advisor prompt with inline sub-agents#34063
Merged
pelikhan merged 3 commits intoMay 22, 2026
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Optimize Daily A/B Testing Advisor workflow
Optimize ab-testing-advisor prompt with inline sub-agents
May 22, 2026
pelikhan
deleted the
copilot/subagent-optimizer-optimize-ab-testing-advisor
branch
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the A/B Testing Advisor workflow prompt to delegate two extractive steps to inline model: small sub-agents, reducing repeated large-file ingestion in the parent prompt while keeping the existing campaign/issue-creation flow in the main agent.
Changes:
- Refactors
.github/workflows/ab-testing-advisor.mdto (a) store a chosen workflow path inSELECTED, (b) delegate workflow characterization to aworkflow-characterizersub-agent, and (c) delegate experiment-field verification to afield-presence-checkersub-agent. - Regenerates workflow lock artifacts and action locks to align pinned SHAs / schedules with their sources (e.g.,
docker/setup-buildx-actionpin and developer-docs consolidator schedule).
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/ab-testing-advisor.md | Introduces SELECTED reuse + adds two inline small sub-agents for workflow characterization and field presence checking. |
| .github/workflows/developer-docs-consolidator.lock.yml | Updates the compiled lock workflow schedule (weekly) and regenerates prompt/config heredocs accordingly. |
| .github/workflows/release.lock.yml | Updates the pinned SHA/version annotation for docker/setup-buildx-action in the compiled workflow. |
| .github/aw/actions-lock.json | Adds a lock entry for docker/setup-buildx-action@v4 matching the updated pin used by the release workflow. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 3
Comment on lines
+118
to
+129
| From the list of workflows **without** an `experiments:` section, pick one at random — **excluding any workflow whose basename appears in the `recently_analyzed` list above** — and store the chosen path in `SELECTED`: | ||
|
|
||
| ```bash | ||
| grep -rL 'experiments:' .github/workflows/*.md 2>/dev/null | grep -v shared | shuf -n 1 | ||
| SELECTED=$(grep -rL 'experiments:' .github/workflows/*.md 2>/dev/null | grep -v shared | shuf -n 1) | ||
| echo "$SELECTED" | ||
| ``` | ||
|
|
||
| If after filtering out recently-analyzed workflows the candidate list is empty, fall back to any eligible workflow (the dedup window has been exhausted): | ||
|
|
||
| ```bash | ||
| grep -rL 'experiments:' .github/workflows/*.md 2>/dev/null | grep -v shared | shuf -n 1 | ||
| SELECTED=$(grep -rL 'experiments:' .github/workflows/*.md 2>/dev/null | grep -v shared | shuf -n 1) | ||
| echo "$SELECTED" |
| gh run list --workflow="$(basename "$SELECTED" .md).lock.yml" --limit 10 --json conclusion,createdAt,displayTitle,durationMS | ||
| ``` | ||
| 7. **Existing quality signals** — Are there any reported issues, quality labels, or patterns in runs? | ||
| Use the `workflow-characterizer` agent with the selected workflow file path. Use the returned characterization (`purpose`, `triggers`, `engine`, `prompt_density`, `tools`, `outputs`, `quality_signals`) as the basis for Step 3. |
Comment on lines
+424
to
+426
| "analysis_type": { "present": "yes", "evidence": "" }, | ||
| "tags": { "present": "yes", "evidence": "" }, | ||
| "notify": { "present": "yes", "evidence": "" } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Daily A/B Testing Advisor was repeatedly pulling large workflow and experiment-source files into the parent model context. This updates the workflow to delegate the two extractive phases to inline
smallsub-agents, reducing redundant context load while preserving the existing decision flow.Primary quest: workflow characterization
workflow-characterizerinline agent.purpose,triggers,engine,prompt_density,tools,outputs, andquality_signals.SELECTEDand reused for run-history lookup.Side quest: experiment field verification
cat pkg/workflow/compiler_experiments.go/cat actions/setup/js/pick_experiment.cjsblocks.field-presence-checkerinline agent that returnspresent/evidenceresults foranalysis_type,tags, andnotify.Inline sub-agent definitions
## agent:blocks following repository convention:workflow-characterizerfield-presence-checkermodel: smalland constrained to factual extraction/classification.Prompt shape
Example of the new delegation pattern: