You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a model-based, stateful property test that drives canary-rollout.sh through randomly generated sequences: candidate cuts, fleet-paused runs, API outages, overrides, promotes and rollbacks. It runs against a fake GitHub API and asserts structural invariants after every step, catching the class of sequence bugs that 535 example-based bats tests keep missing.
Market Signal
Stateful property-based testing (Hypothesis RuleBasedStateMachine, Hegel, PropEr) is the established technique for finding sequence bugs in state machines, using random command generation, invariant checks after each step and shrinking to a minimal failing trace. Progressive-delivery controllers such as Argo Rollouts and Flagger treat 'analysis inconclusive' and 'no traffic' as distinct from failure, which is exactly the class of bug open here.
User Signal
About ten open canary bugs are sequence or structural, not single-input: #1086/#1121 (soak counts runs not executing the candidate), #1310 (fleet-paused run counted healthy), #1090/#1270 (rings structurally unoccupiable; stable frozen since 2026-07-10), #1103 (rollback is a silent no-op for v-scoped agents), #1223 (promote targets a different commit than its dry run), #1324 (two candidates deadlock; API outage becomes permanent REGRESSION), #1244/#1247 follow-ups.
Technical Opportunity
scripts/canary-rollout.sh (4.2k lines) and scripts/lib/canary-rollout.sh already isolate decisions behind gh calls that tests stub, and canary-rings.json is declarative. A Python Hypothesis state machine (pytest is already in CI via pytest-requirements.txt) can model rings, tags and runs and invoke the script with a stubbed gh on PATH. It then checks invariants: every ring is occupiable; only candidate-executing runs count; inconclusive is never healthy and never a regression; rollback moves the tag consumers resolve; promote is bound to the dry-run SHA; no reachable state has no advancing command. The static invariants (occupiability, SC2 contradictions) can also run as a cheap canary-rings.json lint.
Assessment
Dimension
Score
Rationale
Feasibility
med
See Technical Opportunity
Impact
high
See User Signal
Urgency
high
See Market Signal
Adversarial Review
Strongest objection: Duplicating canary-rollout.sh's semantics in a Python model is a second implementation that will drift, and the existing bugs are already filed. Just fix them.
Rebuttal: The model does not reimplement the gate. It only encodes invariants (properties), which are a few dozen lines and change rarely. Every one of the filed bugs would have been a shrunk counterexample, and fixing them one at a time has not stopped new ones (#1324 arrived after #1086). The structural lint alone would have flagged #1090 and #1270 on the PR that introduced them, and stable has been frozen for 4 months because of #1270.
Suggested Next Step
Write a canary-rings.json invariant lint (occupiability plus SC2 consistency) with a failing fixture reproducing #1090/#1270. Then spike a 6-command Hypothesis state machine (cut, run-ok, run-paused, api-fail, promote, rollback) asserting the #1310 and #1103 invariants, and confirm it reproduces both before the fixes land.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Summary
Add a model-based, stateful property test that drives canary-rollout.sh through randomly generated sequences: candidate cuts, fleet-paused runs, API outages, overrides, promotes and rollbacks. It runs against a fake GitHub API and asserts structural invariants after every step, catching the class of sequence bugs that 535 example-based bats tests keep missing.
Market Signal
Stateful property-based testing (Hypothesis RuleBasedStateMachine, Hegel, PropEr) is the established technique for finding sequence bugs in state machines, using random command generation, invariant checks after each step and shrinking to a minimal failing trace. Progressive-delivery controllers such as Argo Rollouts and Flagger treat 'analysis inconclusive' and 'no traffic' as distinct from failure, which is exactly the class of bug open here.
User Signal
About ten open canary bugs are sequence or structural, not single-input: #1086/#1121 (soak counts runs not executing the candidate), #1310 (fleet-paused run counted healthy), #1090/#1270 (rings structurally unoccupiable; stable frozen since 2026-07-10), #1103 (rollback is a silent no-op for v-scoped agents), #1223 (promote targets a different commit than its dry run), #1324 (two candidates deadlock; API outage becomes permanent REGRESSION), #1244/#1247 follow-ups.
Technical Opportunity
scripts/canary-rollout.sh (4.2k lines) and scripts/lib/canary-rollout.sh already isolate decisions behind gh calls that tests stub, and canary-rings.json is declarative. A Python Hypothesis state machine (pytest is already in CI via pytest-requirements.txt) can model rings, tags and runs and invoke the script with a stubbed gh on PATH. It then checks invariants: every ring is occupiable; only candidate-executing runs count; inconclusive is never healthy and never a regression; rollback moves the tag consumers resolve; promote is bound to the dry-run SHA; no reachable state has no advancing command. The static invariants (occupiability, SC2 contradictions) can also run as a cheap canary-rings.json lint.
Assessment
Adversarial Review
Strongest objection: Duplicating canary-rollout.sh's semantics in a Python model is a second implementation that will drift, and the existing bugs are already filed. Just fix them.
Rebuttal: The model does not reimplement the gate. It only encodes invariants (properties), which are a few dozen lines and change rarely. Every one of the filed bugs would have been a shrunk counterexample, and fixing them one at a time has not stopped new ones (#1324 arrived after #1086). The structural lint alone would have flagged #1090 and #1270 on the PR that introduced them, and stable has been frozen for 4 months because of #1270.
Suggested Next Step
Write a canary-rings.json invariant lint (occupiability plus SC2 consistency) with a failing fixture reproducing #1090/#1270. Then spike a 6-command Hypothesis state machine (cut, run-ok, run-paused, api-fail, promote, rollback) asserting the #1310 and #1103 invariants, and confirm it reproduces both before the fixes land.
All reactions