Skip to content

Enable experiment-driven model selection in smoke-copilot workflow#32255

Merged
pelikhan merged 1 commit into
mainfrom
copilot/modify-agentic-workflow-experiment
May 15, 2026
Merged

Enable experiment-driven model selection in smoke-copilot workflow#32255
pelikhan merged 1 commit into
mainfrom
copilot/modify-agentic-workflow-experiment

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 15, 2026

New Feature

What does this feature do?

This PR adds a model A/B experiment to an existing agentic workflow and routes a model field through the selected experiment variant. It validates that workflow model selection can be parameterized by experiments.* values.

Why is this feature needed?

We need explicit coverage for experiment-based model assignment to ensure model configuration is not static and can participate in controlled variant rollouts.

Implementation details

  • Added model experiment dimension

    Introduced a new experiments.subagent_model variant set in .github/workflows/smoke-copilot.md:

    • small
    • large
  • Bound model field to experiment output

    Updated the file-summarizer sub-agent definition to use the experiment-selected model via expression-based assignment.

  • Regenerated compiled workflow

    Recompiled smoke-copilot so the generated lock file reflects experiment wiring and runtime expression expansion.

experiments:
  caveman: [yes, no]
  subagent_model: [small, large]

## agent: `file-summarizer`
---
model: ${{ experiments.subagent_model }}
description: Summarizes the content of a file in a few concise sentences
---

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title Add model experiment variant in smoke-copilot workflow Enable experiment-driven model selection in smoke-copilot workflow May 15, 2026
Copilot AI requested a review from pelikhan May 15, 2026 02:26
@pelikhan pelikhan marked this pull request as ready for review May 15, 2026 02:34
Copilot AI review requested due to automatic review settings May 15, 2026 02:34
@pelikhan pelikhan merged commit fd9d34a into main May 15, 2026
@pelikhan pelikhan deleted the copilot/modify-agentic-workflow-experiment branch May 15, 2026 02:34
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an experiment dimension to route the smoke Copilot sub-agent model through experiment-selected variants.

Changes:

  • Adds subagent_model experiment variants (small, large).
  • Updates file-summarizer model selection to reference the selected experiment value.
  • Regenerates the compiled workflow lock file with experiment wiring.
Show a summary per file
File Description
.github/workflows/smoke-copilot.md Adds the model experiment and wires it into the inline sub-agent definition.
.github/workflows/smoke-copilot.lock.yml Regenerated workflow reflecting the added experiment outputs and prompt environment bindings.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 1

## agent: `file-summarizer`
---
model: small
model: ${{ experiments.subagent_model }}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants