Skip to content

Add prompt_style A/B experiment to blog-auditor workflow#32630

Merged
pelikhan merged 2 commits into
mainfrom
copilot/ab-advisor-experiment-prompt-style
May 16, 2026
Merged

Add prompt_style A/B experiment to blog-auditor workflow#32630
pelikhan merged 2 commits into
mainfrom
copilot/ab-advisor-experiment-prompt-style

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 16, 2026

This updates blog-auditor to run a prompt_style experiment (detailed vs concise) so we can measure whether a goal-oriented prompt reduces token/runtime cost without degrading audit correctness. The workflow prompt now supports compile-time variant selection while preserving the existing detailed path.

  • Experiment configuration (frontmatter)

    • Added experiments.prompt_style with:
      • variants: [detailed, concise]
      • primary/secondary metrics
      • guardrail metrics
      • sample size, weights, start date, analysis type, tags
      • notify.issue set to the created issue number
    • Kept config aligned with current workflow schema (guardrail thresholds as expressions).
  • Prompt variant branching

    • Wrapped audit instructions in a conditional on experiments.prompt_style.
    • Added a concise variant focused on outcomes (navigate, validate checks, create Audits discussion with pass/fail table and remediation).
    • Kept the existing detailed instruction body intact under the else branch.
  • Compiled workflow artifact update

    • Regenerated blog-auditor.lock.yml to reflect the new experiment metadata and prompt conditional rendering.
{{#if experiments.prompt_style == 'concise' }}
## Audit Process
Navigate to the target URL with Playwright, capture accessibility snapshot, validate
status/url/content/keywords/snippets, then create an Audits discussion with a check summary.
{{else}}
## Audit Process
...existing detailed step-by-step instructions...
{{/if}}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add A/B test for prompt style in blog-auditor workflow Add prompt_style A/B experiment to blog-auditor workflow May 16, 2026
Copilot AI requested a review from pelikhan May 16, 2026 14:41
@pelikhan pelikhan marked this pull request as ready for review May 16, 2026 14:42
Copilot AI review requested due to automatic review settings May 16, 2026 14:42
@pelikhan pelikhan merged commit 6cd2c71 into main May 16, 2026
@pelikhan pelikhan deleted the copilot/ab-advisor-experiment-prompt-style branch May 16, 2026 14:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a prompt_style A/B experiment to the blog auditor workflow to compare detailed versus concise audit prompts.

Changes:

  • Adds experiments.prompt_style metadata for detailed/concise variants.
  • Wraps the audit prompt in a variant conditional and introduces a concise audit path.
  • Regenerates the compiled lock workflow with experiment selection/state handling.
Show a summary per file
File Description
.github/workflows/blog-auditor.md Adds experiment metadata and concise/detailed prompt branching.
.github/workflows/blog-auditor.lock.yml Regenerated workflow artifact with experiment state, selection, and propagation steps.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 3

Comment on lines +31 to +32
notify:
issue: 32603
- All required keywords present: `agentic-workflows`, `GitHub`, `workflow`, `compiler`
- Any YAML/Markdown workflow code snippets pass `gh aw compile --no-emit --validate`

Create a discussion in the **Audits** category titled `[audit] Agentic Workflows blog audit - PASSED` (or `FAILED`). Include a summary table of each check with pass/fail status and the values observed. For failures, add suggested remediation steps.
- HTTP status is 200
- Final URL is within `githubnext.com` / `www.githubnext.com`
- Content length exceeds 5,000 characters
- All required keywords present: `agentic-workflows`, `GitHub`, `workflow`, `compiler`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ab-advisor] Experiment campaign for blog-auditor: A/B test prompt_style

3 participants