Skip to content

ci: one runner for the WCM demos, callable from the SDK repository - #92

Merged
imran-siddique merged 1 commit into
mainfrom
feat/wcm-demo-runner
Aug 27, 2026
Merged

ci: one runner for the WCM demos, callable from the SDK repository#92
imran-siddique merged 1 commit into
mainfrom
feat/wcm-demo-runner

Conversation

@imran-siddique

Copy link
Copy Markdown
Member

Closes the gap that let 0.27.0 break six demos here without anything noticing.

Why nothing caught it

The list of demos to run lived inside .github/workflows/ci.yml, where the SDK repository's CI cannot reach it. And this repository only ever tests against a version already on PyPI.

So the earliest possible detection was after publishing, which is after the point where a version number can be taken back.

What changes

run_demos.py moves the list next to the demos, where both repositories call it:

Repository Runs it against Catches
This one The published PyPI package A demo somebody broke
weight-custody-manifest A wheel built from the branch under review An SDK change that breaks the demos, on the PR that causes it

The SDK-side job is a follow-up PR there; this one has to land first so the script exists on main.

Discovered, not listed

Every top-level module that is not a test runs. A new demo is covered the day it lands, which matters because the gap being closed is a change nobody happened to exercise.

A demo that cannot run offline goes in REQUIRES_NETWORK with a reason and is reported as skip on every run, so exclusions stay visible rather than living in a comment nobody re-reads. Two today: real_open_model.py downloads a model, real_lora_custody.py trains a LoRA adapter.

$ python run_demos.py --list
13 demo(s) would run:
  channel_binding.py
  ...
2 skipped:
  real_lora_custody.py: trains a LoRA adapter; needs torch and a download
  real_open_model.py: downloads a real model from the Hugging Face Hub

Two small output choices

Failures print after the summary rather than interleaved. Six demos failing the same way is a different problem from one failing alone, and that is the first thing worth knowing.

Both streams are captured. A demo printing its refusal to stdout and a trace to stderr is the normal shape here, and reading only one has sent people looking in the wrong place.

Verified

all 13 demos and the unit tests pass

And, more usefully, verified it fails correctly: reverting refuse_and_wipe.py to its pre-fix state makes the runner single it out and exit non-zero.

🤖 Generated with Claude Code

https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak

weight-custody-manifest 0.27.0 shipped two correct security changes and both
broke demos here: release began refusing manifests whose identity was not
pinned out of band, and the memory-fingerprint challenge began requiring a
signed sweep. Six demos that passed on 0.26.0 failed on 0.27.0, and nothing
caught it.

Nothing could have. The list of demos to run lived inside this repository's
workflow file, where the SDK's CI cannot reach it, and this repository only
ever tests against a version already on PyPI. So the earliest possible
detection was after publishing, which is after the point where a version number
can be taken back.

run_demos.py moves the list next to the demos, where both repositories can call
it. This one runs it against the published package, catching a demo somebody
broke. The SDK repository runs it against a wheel built from the branch under
review, catching an SDK change that breaks the demos, on the pull request that
causes it.

Demos are discovered rather than listed. Every top-level module that is not a
test runs, so a new demo is covered the day it lands, which matters because the
gap being closed is a change nobody happened to exercise. A demo that cannot
run offline goes in REQUIRES_NETWORK with a reason and is reported as skipped
on every run, so the exclusions stay visible instead of living in a comment
nobody re-reads.

Failures print after the summary rather than interleaved, because six demos
failing the same way is a different problem from one failing alone and that is
the first thing worth knowing. Both streams are captured: a demo printing its
refusal to stdout and a trace to stderr is the normal shape here.

Verified by reverting one demo to its pre-fix state and watching the runner
single it out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak
@imran-siddique
imran-siddique requested a review from a team as a code owner August 27, 2026 16:15
@imran-siddique
imran-siddique merged commit d1ec045 into main Aug 27, 2026
18 checks passed
@imran-siddique
imran-siddique deleted the feat/wcm-demo-runner branch August 27, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant