Skip to content

Add tone-style A/B experiment to Typist workflow#34033

Merged
pelikhan merged 3 commits into
mainfrom
copilot/ab-advisor-experiment-tone-variant
May 22, 2026
Merged

Add tone-style A/B experiment to Typist workflow#34033
pelikhan merged 3 commits into
mainfrom
copilot/ab-advisor-experiment-tone-variant

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 22, 2026

The Typist workflow now supports an A/B experiment on prompt tone to compare the current formal style against a more conversational variant while preserving the existing analysis structure. This wires the experiment into both workflow metadata and prompt rendering so runs can be attributed and compared.

  • Experiment metadata

    • Added a rich experiments.tone_style config to typist.md
    • Captures hypothesis, primary/secondary metrics, guardrails, sample size, weights, start date, tags, and notification target
    • Keeps the experiment self-describing in frontmatter and compiled workflow output
  • Prompt tone variants

    • Wrapped the mission in explicit variant conditionals
    • Added a tone-specific Phase 0 introduction
    • Split the executive summary instructions into formal vs conversational guidance
    • Swapped the recommendations section header/intro based on the selected variant
  • Compiled workflow update

    • Regenerated typist.lock.yml so experiment metadata and conditional prompt rendering are reflected in the compiled workflow
{{#if experiments.tone_style == 'formal'}}
Analyze all Go source files in the repository to identify:
1. **Duplicated type definitions**
2. **Untyped usages**
{{/if}}
{{#if experiments.tone_style == 'conversational'}}
Let's hunt for type consistency issues in the Go codebase!
{{/if}}

Copilot AI and others added 2 commits May 22, 2026 15:12
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add A/B test for tone_variant in typist workflow Add tone-style A/B experiment to Typist workflow May 22, 2026
Copilot AI requested a review from pelikhan May 22, 2026 15:21
@pelikhan pelikhan marked this pull request as ready for review May 22, 2026 16:00
Copilot AI review requested due to automatic review settings May 22, 2026 16:00
@pelikhan pelikhan merged commit fbbf31e into main May 22, 2026
@pelikhan pelikhan deleted the copilot/ab-advisor-experiment-tone-variant branch May 22, 2026 16:01
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an A/B experiment to the Typist workflow to compare a formal vs conversational prompt tone, and wires the selected variant into compiled workflow metadata and prompt rendering so runs can be attributed and analyzed.

Changes:

  • Introduces experiments.tone_style frontmatter in typist.md and adds tone-dependent prompt sections via {{#if experiments.tone_style == ...}} blocks.
  • Updates the compiled typist.lock.yml to restore/pick/persist experiment state, export the selected variant, and include experiment assignments in workflow outputs.
  • Regenerates additional compiled workflow artifacts (smoke-temporary-id.lock.yml, ai-moderator.lock.yml).
Show a summary per file
File Description
.github/workflows/typist.md Declares the tone_style experiment and conditionally swaps tone-specific prompt sections.
.github/workflows/typist.lock.yml Compiled workflow updated to select, propagate, and persist experiment assignments and render the prompt accordingly.
.github/workflows/smoke-temporary-id.lock.yml Regenerated compiled workflow output (non-Typist changes).
.github/workflows/ai-moderator.lock.yml Regenerated compiled workflow output; updates skip-bots env list.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 4/4 changed files
  • Comments generated: 2

Comment on lines +1 to 5
# gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"f53e3dc40b0ce0efcc0d2598bbbfac5c5aaf180ae5eec48b8e0d8cea9c56f62e","strict":true,"agent_id":"copilot"}
# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GH_AW_OTEL_GRAFANA_AUTHORIZATION","GH_AW_OTEL_GRAFANA_ENDPOINT","GH_AW_OTEL_SENTRY_AUTHORIZATION","GH_AW_OTEL_SENTRY_ENDPOINT","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.51"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.51"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.51"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.17"},{"image":"ghcr.io/github/github-mcp-server:v1.0.4"},{"image":"node:lts-alpine","digest":"sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f","pinned_image":"node:lts-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f"}]}
# ___ _ _
# / _ \ | | (_)
# | |_| | __ _ ___ _ __ | |_ _ ___
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
GH_AW_SKIP_BOTS: "github-actions,copilot,dependabot,renovate,github-copilot-enterprise,copilot-swe-agent"
GH_AW_SKIP_BOTS: "github-actions,copilot-swe-agent,Copilot,copilot,dependabot,renovate,github-copilot-enterprise"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ab-advisor] Experiment campaign for typist: A/B test tone_variant

3 participants