Skip to content

fix(experiments): require a winning variation when stopping an iteration#71

Merged
hsadhvani merged 1 commit into
mainfrom
fix/experiment-stop-requires-winner
May 29, 2026
Merged

fix(experiments): require a winning variation when stopping an iteration#71
hsadhvani merged 1 commit into
mainfrom
fix/experiment-stop-requires-winner

Conversation

@hsadhvani
Copy link
Copy Markdown
Contributor

Summary

The launchdarkly-experiment-setup skill told the model it could stop an experiment iteration with "no winner declared." In practice LaunchDarkly requires you to declare a winning variation to stop an iteration, so that guidance led to failed stops.

This updates the skill so that stopping always declares a winningTreatmentId (and winningReason). When an experiment is inconclusive, the guidance is to declare the baseline/control as the winner — there is no "stop without a winner" path.

Paired with a fix to the stop-experiment-iteration MCP tool in gram-functions that makes winningTreatmentId required.

Testing approaches

Skill-only doc change; reviewed against the actual stop-experiment-iteration tool behavior.

Feature flags

N/A

@hsadhvani hsadhvani requested a review from a team as a code owner May 29, 2026 17:51
@github-actions
Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 75/100 (3/4) -25
agentcontrol/configs-update 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (3/3) no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

@hsadhvani hsadhvani enabled auto-merge (squash) May 29, 2026 18:16
@hsadhvani hsadhvani merged commit f85169c into main May 29, 2026
12 checks passed
@hsadhvani hsadhvani deleted the fix/experiment-stop-requires-winner branch May 29, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants