Skip to content

ci: run the live smoke harness as a release gate - #165

Merged
yuanhao merged 1 commit into
mainfrom
ci/live-smoke-gate
Aug 24, 2026
Merged

ci: run the live smoke harness as a release gate#165
yuanhao merged 1 commit into
mainfrom
ci/live-smoke-gate

Conversation

@yuanhao

@yuanhao yuanhao commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Closes #161.

Why

release_smoke found a real bug on its first ever run — tools with zero parameters were uncallable on Anthropic, in shipped releases. Since then it has run when someone remembered.

That is the same failure mode the price audit had before it went into the release checklist: a check that depends on memory is a check you do not have.

The gate

Manual dispatch, plus automatic on release/** push. Both providers run independently — they are separate SSE parsers and tool-call accumulators, and the zero-argument bug lived in exactly that layer.

decision why
Not per-PR CI needs keys, costs money per run, and live calls are flaky — two long_horizon runs hung mid-session during #150
15-minute timeout the harness has none, and a hung provider call should fail the gate rather than block the queue
Missing key skips, not fails a fork without secrets shouldn't see a red X for a gate it can't run
long_horizon excluded it's a diagnostic — slower, and its output wants reading rather than a pass/fail

The harness already exits non-zero on any failed check, so it gates as-is with no changes.

Verified

  • Workflow YAML parses; triggers, matrix, and timeout confirmed
  • SMOKE_MODEL="" (the Anthropic row passes an empty string, not unset) falls through to the Sonnet 5 default — checked, not assumed

Documented in CONTRIBUTING.md and added to the release checklist as step 3b, next to the price audit.

Note on a sibling issue

While writing the CONTRIBUTING section I found that #162's premise was wrong — I'd claimed the SSE parsing layer had no coverage, but tests/anthropic_stream_test.rs has 23 wiremock-based tests, google_stream_test.rs 8, openai_compat_stream_test.rs 3. I've corrected that issue and re-scoped it to the actual gap: empty tool arguments are untested while malformed ones are pinned, and openai_compat has thin coverage for something backing 15+ providers.

CI-and-docs only — no library change.

🤖 Generated with Claude Code

release_smoke found a real bug on its first ever run — tools with zero
parameters were uncallable on Anthropic, in shipped releases. It has run
since when someone remembered, which is the same failure the price audit
had before it went into the checklist: a check that depends on memory is
a check you do not have.

Dispatchable manually, and fires automatically on release/** push. Both
providers report independently — they are separate SSE parsers and
tool-call accumulators, and the zero-argument bug lived in exactly that
layer.

Deliberately not per-PR CI: it needs provider keys, costs money per run,
and live calls are flaky enough that a hung request would block the
queue. Per-release is the right granularity. A 15-minute timeout bounds
it, because the harness has none and two long_horizon runs hung
mid-session during #150 work.

A missing key skips rather than fails, so a fork without secrets does not
see a red X for a gate it cannot run.

long_horizon is deliberately excluded: it is a diagnostic, slower, and
its output wants reading rather than a pass/fail.

Documented in CONTRIBUTING and in the release checklist.

Closes #161
@yuanhao
yuanhao merged commit 3dbe19a into main Aug 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run the live smoke harness as a release gate, not by memory

1 participant