Add prompt_style A/B experiment to daily-news workflow#31192
Merged
Conversation
7 tasks
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Experiment campaign for daily-news: A/B test prompt style
Add May 9, 2026
prompt_style A/B experiment to daily-news workflow
pelikhan
approved these changes
May 9, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a prompt_style A/B experiment to the daily-news agentic workflow to compare the existing verbose prompt against a new concise variant, and regenerates the compiled lock workflow to wire up experiment selection/state persistence.
Changes:
- Introduces an
experiments.prompt_styleconfiguration block (variants + metrics + guardrails) to thedaily-newsworkflow frontmatter. - Adds handlebars conditionals to branch between
detailedandconciseprompt instructions for charts and the report body. - Regenerates
.github/workflows/daily-news.lock.yml, including experiment pick/persist plumbing and other compile-time output changes.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/daily-news.md | Adds experiment metadata + prompt-body branching for detailed vs concise. |
| .github/workflows/daily-news.lock.yml | Recompiled workflow lockfile to include experiment runtime/state jobs and updated generated workflow content. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 4
| variants: [detailed, concise] | ||
| description: "Tests whether a concise directive produces equivalent discussion quality to the current verbose 5-phase prompt" | ||
| hypothesis: "H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score" | ||
| metric: effective_token_count |
Comment on lines
+39
to
+40
| notify: | ||
| issue: 31190 |
| # Fix permissions on firewall logs/audit dirs so they can be uploaded as artifacts | ||
| # AWF runs with sudo, creating files owned by root | ||
| sudo chmod -R a+rX /tmp/gh-aw/sandbox/firewall 2>/dev/null || true | ||
| sudo chmod -R a+r /tmp/gh-aw/sandbox/firewall 2>/dev/null || true |
Comment on lines
66
to
70
| name: "Daily News" | ||
| "on": | ||
| schedule: | ||
| - cron: "34 8 * * 1-5" | ||
| - cron: "45 8 * * 1-5" | ||
| # Friendly format: daily around 9:00 on weekdays (scattered) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Enhancement
The
daily-newsprompt is currently highly prescriptive; this introduces an A/B campaign to measure whether a concise prompt can reduce token usage while preserving output quality and chart/discussion reliability. The workflow now supportsdetailedvsconciseprompt variants with experiment metadata and runtime branching.Experiment configuration (frontmatter)
experiments.prompt_styleobject to.github/workflows/daily-news.md:detailed,conciseeffective_token_countoutput_length_chars,run_duration_ms,chart_generatedmin_samples,weight,start_date,analysis_type,tags.github/workflows/daily-news.lock.ymlvia workflow compile.Prompt-body variant gating
detailedpath keeps existing long-form instructions;concisepath uses a short directive focused on:/tmp/gh-aw/daily-news-data/Lockfile cleanup
actions/github-scriptmanifest comment entry in the generated lock file.{{#if experiments.prompt_style == "concise"}} ## 📊 Trend Charts Requirement Generate exactly **2 trend charts** ... from `/tmp/gh-aw/daily-news-data/` ... {{else}} ## 📊 Trend Charts Requirement **IMPORTANT**: Generate exactly 2 trend charts ... ### Chart Generation Process ... {{/if}}