Run the PBMC 3k tutorials in CI - #99
Merged
Merged
Conversation
test_tutorial_smoke.py ran nowhere but a developer's machine, and #98 is what that cost: a correct guard that never executed, and a release shipped with the platelet cluster captioned "DC" in the tutorial's headline figure. The new `tutorials` job caches the 28 MB PBMC 3k dataset and runs the eleven smoke tests that need only it — six tutorial scripts end to end, plus the marker table, the object-model round trip, the dim-reduc extras and both cell-type-map guards. Two decisions carry the job: * **A skip fails it.** Every selected test needs only the cached dataset, so a skip means a broken cache, a misspelt TRUECELL_TUTORIAL_SMOKE, or a new test wanting a dataset this job does not fetch. All three would otherwise be a green tick on a job that checked nothing — the failure mode being fixed. Verified both ways: unsetting the variable and pointing HOME elsewhere each produce "11 skipped" and each fails the step. * **The filter deselects the eight uncached datasets** rather than selecting pbmc3k by name. An allow-list reads better and would let a pbmc3k test added later under another name quietly never run here. The remaining ~200 MB of datasets stay developer-run, so the full opt-in suite before a release is still the rule; ROADMAP.md and the dev skill say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Local timing for the selected slice: 11 passed in 6m03s.
Three tests are 81% of it. CI on If that is too much per pull request, the natural trim is |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap #98 was found through.
test_tutorial_smoke.pyis opt-in and rannowhere but a developer's machine, so
test_cell_type_map_matches_the_markers—correct, and written precisely to stop pbmc3k figure labels drifting — never
executed, and 1.0.0 shipped with the 14-cell platelet cluster captioned "DC" in
the tutorial's headline UMAP.
The job
A
tutorialsjob that caches the 28 MB PBMC 3k dataset and runs the elevensmoke tests needing only it:
lazy/BPCells
test_pbmc3k_prints_the_marker_table,test_pbmc3k_object_model_round_trips,test_pbmc3k_dimreduc_extras_hold_upThe other eight datasets total ~200 MB and stay developer-run. This is the slice
that pays for itself: it is where the labelled figures come from, and it is the
one that has now gone wrong twice.
Two decisions that carry the job
A skip fails it. Every selected test needs only the cached dataset, so a skip
can only mean a broken cache, a misspelt
TRUECELL_TUTORIAL_SMOKE, or a new testwanting a dataset this job does not fetch. Each would otherwise be a green tick
on a job that verified nothing — which is the exact failure being fixed, and this
repo has a long history of it. Mutation-tested locally, both ways:
11 skipped→ step exits 1HOMEpointed elsewhere (cache miss)11 skipped→ step exits 1A separate
test -f …/matrix.mtxstep catches a half-successful download beforepytest is reached, and pytest's exit 5 covers "collected nothing" if someone
renames a test out of the filter.
The filter deselects the eight uncached datasets rather than selecting pbmc3k
by name. An allow-list (
-k "pbmc3k or cell_type") reads better and is the wrongchoice — a pbmc3k test added later under another name would quietly never run
here, reproducing the defect this job exists to prevent. Inverted, new pbmc3k
tests are picked up automatically, and a test needing an uncached dataset turns
the job red until someone extends the list. Loud, and the right way round.
Verified both forms select the same 11 of 26 today.
Notes
truecell-data-pbmc3k-v1) rather than a hash oftruecell/datasets.py, which holds every other downloader too and woulddiscard the cache on unrelated edits. Bump it by hand if
_PBMC3K_URLSchanges — a comment in the workflow says so.
cf.10xgenomics.com, so an outage thereturns it red. Kept as its own step so that reads as a failed download rather
than a mysterious tutorial failure. Making it advisory instead was tempting and
rejected: a job that cannot fail is what put us here.
ci.yml.estimate in the changelog, since the local slice was still running when this
was opened.
ROADMAP.mdandskills/truecell-devare updated to say the gate is closed forpbmc3k only, and that the full opt-in suite before a release is still the
rule rather than a formality.
🤖 Generated with Claude Code