ci: one runner for the WCM demos, callable from the SDK repository - #92
Merged
Conversation
weight-custody-manifest 0.27.0 shipped two correct security changes and both broke demos here: release began refusing manifests whose identity was not pinned out of band, and the memory-fingerprint challenge began requiring a signed sweep. Six demos that passed on 0.26.0 failed on 0.27.0, and nothing caught it. Nothing could have. The list of demos to run lived inside this repository's workflow file, where the SDK's CI cannot reach it, and this repository only ever tests against a version already on PyPI. So the earliest possible detection was after publishing, which is after the point where a version number can be taken back. run_demos.py moves the list next to the demos, where both repositories can call it. This one runs it against the published package, catching a demo somebody broke. The SDK repository runs it against a wheel built from the branch under review, catching an SDK change that breaks the demos, on the pull request that causes it. Demos are discovered rather than listed. Every top-level module that is not a test runs, so a new demo is covered the day it lands, which matters because the gap being closed is a change nobody happened to exercise. A demo that cannot run offline goes in REQUIRES_NETWORK with a reason and is reported as skipped on every run, so the exclusions stay visible instead of living in a comment nobody re-reads. Failures print after the summary rather than interleaved, because six demos failing the same way is a different problem from one failing alone and that is the first thing worth knowing. Both streams are captured: a demo printing its refusal to stdout and a trace to stderr is the normal shape here. Verified by reverting one demo to its pre-fix state and watching the runner single it out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap that let 0.27.0 break six demos here without anything noticing.
Why nothing caught it
The list of demos to run lived inside
.github/workflows/ci.yml, where the SDK repository's CI cannot reach it. And this repository only ever tests against a version already on PyPI.So the earliest possible detection was after publishing, which is after the point where a version number can be taken back.
What changes
run_demos.pymoves the list next to the demos, where both repositories call it:weight-custody-manifestThe SDK-side job is a follow-up PR there; this one has to land first so the script exists on
main.Discovered, not listed
Every top-level module that is not a test runs. A new demo is covered the day it lands, which matters because the gap being closed is a change nobody happened to exercise.
A demo that cannot run offline goes in
REQUIRES_NETWORKwith a reason and is reported asskipon every run, so exclusions stay visible rather than living in a comment nobody re-reads. Two today:real_open_model.pydownloads a model,real_lora_custody.pytrains a LoRA adapter.Two small output choices
Failures print after the summary rather than interleaved. Six demos failing the same way is a different problem from one failing alone, and that is the first thing worth knowing.
Both streams are captured. A demo printing its refusal to stdout and a trace to stderr is the normal shape here, and reading only one has sent people looking in the wrong place.
Verified
And, more usefully, verified it fails correctly: reverting
refuse_and_wipe.pyto its pre-fix state makes the runner single it out and exit non-zero.🤖 Generated with Claude Code
https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak