Skip to content

Run the PBMC 3k tutorials in CI - #99

Merged
shanikawm merged 1 commit into
mainfrom
ci/cache-pbmc3k-dataset
Aug 7, 2026
Merged

Run the PBMC 3k tutorials in CI#99
shanikawm merged 1 commit into
mainfrom
ci/cache-pbmc3k-dataset

Conversation

@shanikawm

Copy link
Copy Markdown
Contributor

Closes the gap #98 was found through. test_tutorial_smoke.py is opt-in and ran
nowhere but a developer's machine, so test_cell_type_map_matches_the_markers
correct, and written precisely to stop pbmc3k figure labels drifting — never
executed, and 1.0.0 shipped with the 14-cell platelet cluster captioned "DC" in
the tutorial's headline UMAP.

The job

A tutorials job that caches the 28 MB PBMC 3k dataset and runs the eleven
smoke tests needing only it:

  • six tutorial scripts end to end — guided, SCTransform, dim-reduc, objects, DE,
    lazy/BPCells
  • test_pbmc3k_prints_the_marker_table, test_pbmc3k_object_model_round_trips,
    test_pbmc3k_dimreduc_extras_hold_up
  • both cell-type-map guards, including the one added in Label the platelet cluster as platelets, not DC #98

The other eight datasets total ~200 MB and stay developer-run. This is the slice
that pays for itself: it is where the labelled figures come from, and it is the
one that has now gone wrong twice.

Two decisions that carry the job

A skip fails it. Every selected test needs only the cached dataset, so a skip
can only mean a broken cache, a misspelt TRUECELL_TUTORIAL_SMOKE, or a new test
wanting a dataset this job does not fetch. Each would otherwise be a green tick
on a job that verified nothing — which is the exact failure being fixed, and this
repo has a long history of it. Mutation-tested locally, both ways:

Simulated fault Result
Opt-in variable unset (misspelt env var) 11 skippedstep exits 1
HOME pointed elsewhere (cache miss) 11 skippedstep exits 1

A separate test -f …/matrix.mtx step catches a half-successful download before
pytest is reached, and pytest's exit 5 covers "collected nothing" if someone
renames a test out of the filter.

The filter deselects the eight uncached datasets rather than selecting pbmc3k
by name. An allow-list (-k "pbmc3k or cell_type") reads better and is the wrong
choice — a pbmc3k test added later under another name would quietly never run
here, reproducing the defect this job exists to prevent. Inverted, new pbmc3k
tests are picked up automatically, and a test needing an uncached dataset turns
the job red until someone extends the list. Loud, and the right way round.
Verified both forms select the same 11 of 26 today.

Notes

  • The cache key is static (truecell-data-pbmc3k-v1) rather than a hash of
    truecell/datasets.py, which holds every other downloader too and would
    discard the cache on unrelated edits. Bump it by hand if _PBMC3K_URLS
    changes
    — a comment in the workflow says so.
  • On a cache miss the job downloads from cf.10xgenomics.com, so an outage there
    turns it red. Kept as its own step so that reads as a failed download rather
    than a mysterious tutorial failure. Making it advisory instead was tempting and
    rejected: a job that cannot fail is what put us here.
  • No action versions changed; the job reuses the ones already in ci.yml.
  • Runtime will be whatever this PR's own run reports — I have not put an
    estimate in the changelog, since the local slice was still running when this
    was opened.

ROADMAP.md and skills/truecell-dev are updated to say the gate is closed for
pbmc3k only, and that the full opt-in suite before a release is still the
rule rather than a formality.

🤖 Generated with Claude Code

test_tutorial_smoke.py ran nowhere but a developer's machine, and #98 is what
that cost: a correct guard that never executed, and a release shipped with the
platelet cluster captioned "DC" in the tutorial's headline figure.

The new `tutorials` job caches the 28 MB PBMC 3k dataset and runs the eleven
smoke tests that need only it — six tutorial scripts end to end, plus the marker
table, the object-model round trip, the dim-reduc extras and both cell-type-map
guards.

Two decisions carry the job:

* **A skip fails it.** Every selected test needs only the cached dataset, so a
  skip means a broken cache, a misspelt TRUECELL_TUTORIAL_SMOKE, or a new test
  wanting a dataset this job does not fetch. All three would otherwise be a
  green tick on a job that checked nothing — the failure mode being fixed.
  Verified both ways: unsetting the variable and pointing HOME elsewhere each
  produce "11 skipped" and each fails the step.
* **The filter deselects the eight uncached datasets** rather than selecting
  pbmc3k by name. An allow-list reads better and would let a pbmc3k test added
  later under another name quietly never run here.

The remaining ~200 MB of datasets stay developer-run, so the full opt-in suite
before a release is still the rule; ROADMAP.md and the dev skill say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shanikawm

Copy link
Copy Markdown
Contributor Author

Local timing for the selected slice: 11 passed in 6m03s.

Test Time Share
lazy_bpcells_tutorial.py 135.8s 37%
pbmc3k_de_tutorial.py 101.9s 28%
pbmc3k_sctransform_tutorial.py 56.5s 16%
pbmc3k_prints_the_marker_table 14.9s
pbmc3k_tutorial.py 14.2s
cell_type_map_covers_exactly_the_clusters_produced 12.6s
pbmc3k_dimreduc_tutorial.py 7.9s
pbmc3k_dimreduc_extras_hold_up 7.9s
cell_type_map_matches_the_markers 7.4s
pbmc3k_objects_tutorial.py 2.8s
pbmc3k_object_model_round_trips 2.0s

Three tests are 81% of it. CI on ubuntu-latest will be slower than this
machine; the job's own run in this PR is the figure to trust.

If that is too much per pull request, the natural trim is lazy_bpcells and
pbmc3k_de — 4 minutes of the 6, and neither draws a labelled figure, which is
the drift this job exists to catch. Dropping both leaves every figure-producing
tutorial plus both cell-type guards at roughly 2 minutes. I have not done it:
they are honest end-to-end coverage of code no unit test exercises the same way,
and out-of-core in particular has a history here (T-lazy). Happy to trim on
request.

@shanikawm
shanikawm merged commit ba4f7de into main Aug 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant