ci: run the live smoke harness as a release gate - #165
Merged
Conversation
release_smoke found a real bug on its first ever run — tools with zero parameters were uncallable on Anthropic, in shipped releases. It has run since when someone remembered, which is the same failure the price audit had before it went into the checklist: a check that depends on memory is a check you do not have. Dispatchable manually, and fires automatically on release/** push. Both providers report independently — they are separate SSE parsers and tool-call accumulators, and the zero-argument bug lived in exactly that layer. Deliberately not per-PR CI: it needs provider keys, costs money per run, and live calls are flaky enough that a hung request would block the queue. Per-release is the right granularity. A 15-minute timeout bounds it, because the harness has none and two long_horizon runs hung mid-session during #150 work. A missing key skips rather than fails, so a fork without secrets does not see a red X for a gate it cannot run. long_horizon is deliberately excluded: it is a diagnostic, slower, and its output wants reading rather than a pass/fail. Documented in CONTRIBUTING and in the release checklist. Closes #161
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #161.
Why
release_smokefound a real bug on its first ever run — tools with zero parameters were uncallable on Anthropic, in shipped releases. Since then it has run when someone remembered.That is the same failure mode the price audit had before it went into the release checklist: a check that depends on memory is a check you do not have.
The gate
Manual dispatch, plus automatic on
release/**push. Both providers run independently — they are separate SSE parsers and tool-call accumulators, and the zero-argument bug lived in exactly that layer.long_horizonruns hung mid-session during #150long_horizonexcludedThe harness already exits non-zero on any failed check, so it gates as-is with no changes.
Verified
SMOKE_MODEL=""(the Anthropic row passes an empty string, not unset) falls through to the Sonnet 5 default — checked, not assumedDocumented in
CONTRIBUTING.mdand added to the release checklist as step 3b, next to the price audit.Note on a sibling issue
While writing the CONTRIBUTING section I found that #162's premise was wrong — I'd claimed the SSE parsing layer had no coverage, but
tests/anthropic_stream_test.rshas 23 wiremock-based tests,google_stream_test.rs8,openai_compat_stream_test.rs3. I've corrected that issue and re-scoped it to the actual gap: empty tool arguments are untested while malformed ones are pinned, andopenai_compathas thin coverage for something backing 15+ providers.CI-and-docs only — no library change.
🤖 Generated with Claude Code