Skip to content

v0.7.3

Choose a tag to compare

@lucapinello lucapinello released this 15 Aug 15:38
· 32 commits to main since this release
9ad4cce

Chorus v0.7.3

No percentile changes. No background null, region set, retention rule or default fold moved,
so effect and activity percentiles are directly comparable with 0.7.2. What changed is enforcement,
report rendering and two real defects — see the banner on each entry.

Background artefacts. Unchanged: this release is verified against dataset revision
backgrounds-2026-08-12-cherimoya-fold0, the same pair 0.7.2 shipped with, and
_HF_REVISION still pins it. Nothing was rebuilt or re-uploaded.

⚠ Git history was rewritten for this release

Every commit sha changed. If you have a clone, fork, or a pinned sha, read this.

git history carried 142.6 MB of artefacts that no longer exist in the tree and never needed to:
107.2 MB of raw audit output (screenshots, per-probe logs, executed notebooks, two prediction .npz
dumps), a 33.7 MB annotations/gencode.v48.basic.annotation.gtf.gz that chorus setup downloads
anyway, and 1.7 MB of chorus_mcp_output/ that was committed by accident. Purged with
git-filter-repo; the pack went 266.5 MiB → 218.5 MiB and .git 597 MB → 220 MB.

The rewrite alone did not change what GitHub reports, because 27 stale remote branches still
pointed at pre-rewrite commits and kept those objects reachable. The 18 whose PRs were merged were
deleted on 2026-08-14 — their work is in main by construction — and the 8 with no merged PR were
left alone. GitHub recomputes repository size on its own GC schedule, so the published figure lags
the deletion rather than following it.

What this means for you:

  • Re-clone, or hard-reset. git fetch --tags --force && git reset --hard origin/main. A plain
    git pull will report divergent histories, because it is one.
  • Any sha you recorded — in a paper, a notebook, a lockfile, an issue — no longer resolves. Tags
    are the stable reference: all 16 were re-pointed at the rewritten equivalents of the same commits.
  • Nothing in the released tree changed. Verified rather than asserted: HEAD's tree hash is
    byte-identical before and after (b3069df3…), and none of the 338 purged paths was tracked at
    HEAD. The full suite passes on the rewritten history — 1,972 fast, and the release numbers below
    were re-measured on the tagged commit.
  • The purged artefacts still exist, in a full pre-rewrite mirror held by the maintainer. Ask
    rather than assuming they are gone; audits/README.md records this.

Deliberately not purged, though it is where the remaining weight is: 603 MB of superseded HTML
report versions and 125 MB of old notebook copies. Those are the history of files the project still
ships, and several audit reports cite them as evidence.

Fixed

  • 🔐 The guard added to catch the leaked credential was itself carrying it
    (#197).
    tests/test_no_committed_credentials.py
    held the real LDlink token verbatim as a fails-without-fix fixture, and listed itself in
    ALLOW_PATHS so the scan skipped it and passed — so the value redacted from the audit report in
    #188 was put straight back into the repo by the fix for that leak, in the one file guaranteed not
    to be read. The fixture is synthetic now (same shape: 12 bare hex after the word "token"), and
    ALLOW_PATHS is permanently empty: a security guard that exempts a file from its own scan
    cannot report a credential pasted into that file. The guard keeps only a contextual exemption, so
    every prefixed pattern still applies to it.

  • 🔐 That guard only read 655 of 869 tracked files
    (#196).
    It filtered by an allowlist of "text"
    suffixes and so never opened 59 .log files or 20 .html reports — the likeliest hiding
    places, since a log captures whatever was in the environment and a rendered report captures
    whatever was in a URL. Swept those 214 files by hand first (0 hits, so nothing was missed), then
    inverted the default: scanning is the behaviour now and only known-binary suffixes are skipped
    (759 of 869). Coverage is asserted by the test, because a shrinking allowlist is how a guard
    quietly stops guarding.

  • A lab-internal absolute path shipped inside the package
    (#197).
    _find_mamba() hardcoded
    /data/pinello/SHARED_SOFTWARE/miniforge3/bin/mamba ahead of the user's own ~/miniforge3, so
    on that one cluster it silently overrode whatever install they had chosen, and everywhere else it
    was dead code advertising a private filesystem layout. Removed; _find_mamba now honours
    MAMBA_EXE first — the variable mamba's own shell hook exports, which EnvironmentManager
    already honoured and this function did not. That covers shared installs properly, for everyone.

  • The citation did not parse (#197). The BibTeX
    author field separated names with commas; BibTeX's separator is and, and commas within a
    name are the von Last, Jr, First form, which permits at most two. Three commas is a parse error
    or four authors silently collapsed into one — in someone else's paper. It was also the only README
    block no test exercised. Now and-separated in Last, First form, with a version field;
    docs/THIRD_PARTY.md no longer offers a second, differently-worded citation; and CITATION.cff
    is added so GitHub renders a "Cite this repository" button. A test cross-checks the two against each
    other and against chorus.__version__.

  • The MCP quick-start dropped the timeout every other config sets
    (#197).
    claude mcp add … omitted
    CHORUS_NO_TIMEOUT while every JSON config in the README and the shipped .mcp.json set it, and
    the README calls it "recommended for interactive use". That step targets users who skip the code —
    laptop and CPU users — whose first long question could then die on the 300 s predict timeout inside
    a Claude conversation, looking like a broken tool. All three recipes now pass
    -e CHORUS_NO_TIMEOUT=1.

  • 🔐 A live LDlink API token had been committed in plaintext since April and is now redacted
    (#188).
    It sat in a tracked audit report, inside
    a paragraph headed "Reminder / hygiene" that asserted "No copy was written to any on-disk
    location by me during the audit" — the HuggingFace token on the next line was redacted; this
    one was not. The token has been revoked. Redaction does not undo exposure: the value remains in
    this repository's git history, so anyone who mirrored the repo should treat it as compromised
    rather than fixed.

    The reason it survived four secret sweeps is the part worth carrying forward. Every sweep, this
    release's own audit included, searched for prefixed patterns — hf_…, AKIA…. An LDlink
    token is twelve bare hex characters with no prefix, so none of them could ever have matched, and
    each clean result was reported as "no secrets" when it only ever meant "no secrets of the shapes
    we grep for
    ". tests/test_no_committed_credentials.py now covers the unprefixed case
    contextually — a hex or base62 run within 40 characters of token/secret/api-key/password — and is
    verified to flag the exact line that leaked while leaving this repo's thousands of legitimate
    hashes alone. audits/AUDIT_CHECKLIST.md §16 now says to run that test rather than hand-roll a
    grep.

  • chorus health could report an oracle Healthy while checking nothing, and an unreachable
    mamba root was reported as a missing Python interpreter.
    Two defects in the same area, both
    found by work that started as documentation tidying
    (#189,
    #192).

    The dependency probes behind chorus health and validate_environment are near-duplicate dicts
    that had drifted to three different contents — the runner's was missing cherimoya and
    alphagenome_pt, the manager's was missing alphagenome_pt. Because dependencies.get(oracle, []) turns a missing key into "nothing to check", chorus health --oracle alphagenome_pt printed
    "✓ Healthy" without importing a thing. Both completed, and
    tests/test_registries_cover_every_oracle.py now fails with the site and the consequence named.

    Separately, get_environment_info() looked an environment up, obtained its path, and then ran a
    second subprocess purely to fill a packages field — using mamba list -n <name>, which resolves
    the name against MAMBA_ROOT_PREFIX where discovery via env list does not. When the root did
    not contain the env, discovery succeeded (so environment_exists() correctly said True) while
    that call exited 1, and the CalledProcessError handler returned None — which all eleven
    callers
    read as "the environment does not exist". The symptom was get_python_executable()
    reporting "Python executable not found in environment" for an env whose interpreter was exactly
    where it belonged. packages has one consumer, a count in chorus list --verbose. It now resolves
    by -p <path> and a listing failure is non-fatal, and the message names the wrong-root
    possibility rather than asserting the unlikely one.

  • chorus config data-dir overstated the HuggingFace cache by 2×, reporting 41.5 GB against a
    real 20.7 GB (#190). The hub stores each file once
    under <repo>/blobs/<sha> and exposes it as a symlink from <repo>/snapshots/<rev>/<name>, so
    both endpoints sit inside the walked tree and every blob was counted twice. Fixed by deduping
    on (st_dev, st_ino), which is du -L semantics and covers hardlinks too. Note for anyone
    tempted by the one-line version: skipping symlinks drops genomes from 3.3 GB to 170 bytes,
    because those entries point at FASTAs stored outside the tree. All five buckets now match
    du -sL --apparent-size.

  • chorus-mcp accepted and ignored every flag. Its entry point was if "--help" in sys.argv
    followed by mcp.run(), so chorus-mcp --port 9000 silently started a stdio server instead of
    erroring (#190). It now has a real parser and
    exits 2 on an unknown flag. The tool list was hand-maintained and had drifted twice — it announced
    22 against 24 registered — so it is derived from the server now. CHORUS_MCP_OUTPUT_DIR (absent
    from every doc, and defaulting relative to the client's working directory), CHORUS_MCP_DEBUG
    and the FASTMCP_* transport settings are documented, and CHORUS_NO_TIMEOUT no longer
    understates itself — it disables model-load timeouts too.

  • A pip install was missing two runtime dependencies. setup.py builds install_requires
    from requirements.txt, which did not list pyyaml (imported at module level by
    chorus/core/environment/manager.py) or requests (by chorus/utils/annotations.py, working
    only because huggingface_hub happens to pull it in). Both declared, with
    tests/test_declared_dependencies_cover_the_imports.py enforcing the property
    (#188 — this one predates the post-release batch
    and was simply undescribed; #189 touched none of those files).

  • The MCP end-to-end test had been skipping for months, and failed the moment it ran. Its gate
    read only HF_TOKEN/HUGGING_FACE_HUB_TOKEN, while both AlphaGenome load paths try the stored
    credential first — so it skipped on any normally-authenticated host. Widened to
    huggingface_hub.get_token(), which immediately exposed a hardcoded MAMBA_ROOT_PREFIX fallback
    of ~/.local/share/mamba: wrong on any host whose envs live under miniforge3/envs, and the
    cause of the misleading interpreter error above. Now derived from the mamba binary chorus itself
    resolved. The integration suite goes 146 → 148 passing (145 was the figure before #188 added one of its own; 146 is what this batch started from)
    (#190).

  • layers_affected came back in a different order every run. discover_variant_effects built it
    as list({...}) over a set of strings, so its order followed per-process hash(str). It reaches
    MCP replies rather than the committed examples, so it is a separate defect from the F8 drift below
    — and free to remove. Now sorted(). In the same pass, AnalysisRequest.generated_at honours
    SOURCE_DATE_EPOCH, so a regeneration can be diffed at all: previously every committed example
    showed at least a changed timestamp even when nothing numeric moved, which made the F8
    investigation harder than it needed to be
    (#191).

  • chorus cleanup can now reclaim the HuggingFace cache, via a new --hf-cache. Deliberately
    not part of --all: with HF_HOME set, that directory is the HF home and holds the token
    from huggingface-cli login, so --all removing it would destroy a credential and any
    non-chorus model cache. --all now says out loud that it left the cache. This also replaces a
    recipe added in 0.7.3's own install-docs pass that would have deleted exactly that directory
    (#190).

  • ⚠ device='cuda:N' no longer takes a GPU you were not given — on either load path. Six oracle templates (the use_environment=True subprocess path) set CUDA_VISIBLE_DEVICES straight from the ordinal, and so did ChromBPNetOracle._load_direct and Enformer's in-process load — which is the create_oracle default. Under a scheduler granting CUDA_VISIBLE_DEVICES=4,5, cuda:1 means the second GPU I was granted (physical 5); overwriting the mask sent the process to physical GPU 1, somebody else's job. Not hypothetical: GPU 4 was another tenant's throughout the audit that found it.

    Worse on the PyTorch side, where it was a live crash rather than a hazard: Cherimoya's templates masked to N and then handed the same cuda:N string to torch, which indexes within the now-one-device visible set, so every ordinal but 0 died with CUDA error: invalid device ordinal. A documented parameter was simply broken.

    The two frameworks need different handling and that is the point: torch resolves the ordinal itself, so the assignment is gone; TensorFlow selects through the mask, so cuda:N is remapped by one shared chorus.core.platform.resolve_visible_ordinal, which raises naming what N indexes when it is out of range. Verified under CUDA_VISIBLE_DEVICES=2,3: cuda:0 → physical 2, cuda:1 → physical 3.

    The direct path was missed by the first fix and by its own guard, which enumerated only *_source/templates/*template.py — the search was scoped to what the checklist item named rather than to the property. The guard now covers chorus/oracles/*.py too and is mutation-tested. The eight builders under scripts/ were swept in the same pass and were already correct. Both halves are audit findings F1 and G1, in audits/2026-08-12_post_v0.7.2_audit.md and audits/2026-08-13_v0.7.3_release_audit.md.

  • extract_sequence_with_padding no longer returns a short sequence near a chromosome end. Its wide-interval branch handed an out-of-bounds end straight to pysam, which clamps silently, and the metadata it returned hardcoded leftN/rightN = 0 — reporting that no padding was needed. Measured on chr1: 2,114 bp requested 40 bp from the end returned 40 bp. The narrow-interval branch always padded correctly, so the two disagreed about the same question; both are now pinned to agree. Sei reaches this path with total_length=SEI_WINDOW, so a short one-hot could reach the model. Audit F2.

  • "Reference FASTA required" now says what to do about it. Ten sites across nine oracles raised it in two wordings that differed only by punctuation, and neither named the reference_fasta kwarg or chorus genome download hg38. One helper now carries both. Audit F3.

  • The Cell Type column no longer repeats the track label. CHIP:CEBPB:IMR-90 beside a column reading IMR-90 said it twice in 13 of 20 committed reports, up to 22 rows in one. Blanked only on an exact case-insensitive match of the label's final colon-delimited component, so DNASE:fibroblast of lung beside the same string is suppressed while CHIP:CEBPB:HepG2_treated beside HepG2 is not, and Enformer's opaque ids keep the column that carries all their meaning. All reports regenerated; no score moved. Audit F7.

  • Attribution gaps: igv.min.js bundles DOMPurify (Cure53) and pako, both shipped in the wheel and credited nowhere; both are now in docs/THIRD_PARTY.md with the licence notices that survive inside the bundle. Borzoi was described as 7,612 tracks in a docstring where it is 7,611. And THIRD_PARTY.md now states explicitly that EPInformer-seq is first-party — no vendored upstream code, weights from this project's own HuggingFace repo — because "7 of 8 oracles attributed" reads like a missing attribution otherwise. Audit F4–F6.

  • Reports no longer resolve their genome through igv.org's hosted registry (#139). genome: "hg38" reads like a setting but is a registry lookup: igv.js resolves the string against its catalogue and follows the result, so every shipped report opened six remote resources across two hosts — the catalogue, chromosome aliases, hg38.chrom.sizes, cytoBandIdeo.txt.gz, ncbiRefSeq.txt.gz and ranged reads of hg38.2bit. Fourteen requests, and the catalogue fetch is fatal: with the network cut the panel did not degrade, it never appeared. The docstring claiming that inlining igv.min.js made reports "viewable offline, on air-gapped hosts" was therefore false in the strongest available sense.

    Chromosome lengths and the ideogram now come from one vendored 6.1 kB table (UCSC's cytoBandIdeo, primary chromosomes only — the per-chromosome maximum band end is the chromosome length, verified against the FASTA index at chr1 = 248,956,422 both ways), and the gene track from chorus's own GENCODE v48 annotation scoped to the drawn window, which also means the panel's genes agree with the gene names printed beside them. Measured on one report:

    requests hosts load to paint with no network
    before 14 2 9.6 s fatal (genomes.json)
    after 9 1 2.2 s fatal (hg38.2bit)
    after, sequence self-hosted same-origin 0 0 0.8 s works

    Cost: +46.7 kB per report, 2.8% of a 1.65 MiB one.

    The reference sequence is the one thing that cannot be bundled, and that is igv.js's rule rather than a choice. Every version requires a sequence source: omit it and 3.1.1 dies in Ec.loadAll on undefined.startsWith, while 3.8.5 dies on url must be either a 'File', 'string', 'function', or 'Promise'. A data: URI does not substitute either — igv decodes data URIs inline and treats them as a non-indexed FASTA, taking chromosome lengths from the body, so a stub declaring the real lengths in its index renders a perfect ideogram and ruler while every feature track silently draws nothing (3 of 5 canvases painted, against 5 of 5 with a real reference). hg38 is 3 GB.

    For a genuinely air-gapped site, CHORUS_IGV_SEQUENCE_URL points the report at a self-hosted copy, and it accepts a FASTA as well as a 2bit because every chorus install already downloads hg38.fa:

    # serve the report beside the genome — same origin, no CORS, no internet
    # `chorus config data-dir` prints a human-readable block, so resolve the path itself:
    GENOMES=$(python -c 'from chorus.core.globals import describe_layout; print(describe_layout()["genomes"])')
    cp report.html "$GENOMES/" && python -m http.server -d "$GENOMES" 8000

    Same-origin matters: a page opened from file:// has origin null and cannot read a served FASTA, and a second port needs CORS headers — both measured, both fail with Access to XMLHttpRequest ... has been blocked.

    All three render paths (_igv_report, multi_oracle_report, causal) now share one config builder, since three copies of the same dict is precisely how the display-pooling defect shipped with only one of them fixed. Every committed report was regenerated; a test reads the artefacts rather than the generator, because regeneration is what makes the fix real.

  • Three of the nine advertised README recipes could not be run (#202). The variant-effect recipe passed alleles=['A', 'G', 'C', 'T'] at chr11:5247500, where hg38 has C. strict_ref=True is the default and deliberate (#128), so this raises ReferenceAlleleMismatchError rather than substituting — and because it raises before binding variant_effects, the two later recipes that consume that variable died with NameError. The identical block in the TLDR had been corrected one commit earlier and this copy, 600 lines further down, was missed; the guard now checks every allele list in the file against the FASTA rather than the one that was reported. Three further claims in the same walkthrough were wrong rather than fatal: the first value a new user prints, wt['ENCFF413AHU'].values.mean(), is a mean over Enformer's full 114,688 bp output (896 bins × 128 bp) and not the 1 kb queried; the "30-second taste" measures 64 s on an idle H100 and 59 s CPU-only — near-identical, because the cost is TensorFlow import and model load rather than the forward pass, so the quickstart needs no GPU at all; and chorus setup --oracle <name> is 13.7 GiB, not the 85 GiB of the default install, which the "if you only need one oracle" pointer had never quantified. The README also promised 1-based inclusive coordinates "everywhere" while its own first snippet used the tuple form, which is 0-based end-exclusive: measured against hg38, 'chr11:5247500-5247504' yields CATCA and ('chr11', 5247500, 5247504) yields ATCA. Harmless inside a 114 kb window, not harmless at 1 bp resolution.

  • The HuggingFace gate read as global and is per-oracle (#202). Bare chorus setup halts before building anything when no token resolves, which is correct — failing on the last of nine oracles after 85 GiB would be worse. But the only escape hatch documented was --no-weights, which skips the gate by downloading no model weights, so a reader who took it could not run a single snippet on the page. chorus/cli/main.py:93 scopes the prompt to requests that actually include alphagenome, so chorus setup --oracle enformer never asks for a token and produces a fully working ~14 GiB install that runs the entire TLDR. Now stated where the gate is described, with --no-weights relabelled as env provisioning, and pinned by an AST guard so the scoping cannot quietly become global.

  • The README's normalization example raised on its second line (#200). norm.activity_percentile("alphagenome", track_id, ref_value=512.0) — there is no ref_value keyword; the third parameter is raw_signal and is positional, so the documented call raised TypeError: got an unexpected keyword argument. Nothing caught it because no test had ever executed a README snippet. The same block documented an effect percentile of 0.962 against a measured 0.9811 — it moved in the 2026-08 background rebuild and the prose did not follow — and the stratification table described the null with every fraction exactly 2× reality and no cCRE stratum at all, while DEFAULT_REGION_STRATA puts half the null inside cCREs; renormalising the remainder is what doubled the rest. That table is the methods description behind every percentile chorus reports, and it is now pinned to the source of truth.

  • Five published numbers were wrong, and the two guards over them could not fail (#209, #211). Found by an adversarial review of this release's own work rather than by the suite. The single-oracle install size charged 1.87 GiB for Enformer weights because the measuring script globbed *enformer* over the HF cache and the dev host holds two mirrors — lucapinello/chorus-enformer, which oracles/enformer.py:121 actually loads (0.94 GiB), and an unrelated EleutherAI/enformer-official-rough. The README's own per-asset table said ~960 MB two hundred lines below, so the file contradicted itself; the total is 12.81 GiB, and a single-oracle install is "~13 GiB, not 85". The activity-null floor was documented as ~29,000 positions, which is LegNet's value read as the global minimum — measured across all nine NPZs it is 19,504–319,642. The prose under the stratification table still said "the 15 % uniform tail" where the table says 7.5 %, exactly double, left behind when the doubled fractions were corrected. And the worked example's activity percentile was quoted as 0.81 in the intro against a measured 0.9625 in the backgrounds section.

    The guards were the more interesting half. test_the_stratification_table_matches_the_code took a flat 1400-character slice that ran ~700 characters past the table into prose reading "50 % of Enformer's accessibility rows exceeded their own track's null maximum" — so deleting the cCRE stratum outright, the exact regression its own docstring names, passed green; tss_near likewise passed on tss_far's identical 10 %. test_the_documented_calls_use_the_real_signatures matched {method}\([^)]*?(\w+)=, which cannot cross the ) of the nested abs(0.45), so it found zero keywords and checked nothing: renaming the real signed parameter left it green. Both repaired, both verified to fail on the defect they exist to catch, and four further vacuous assertions were found and fixed the same way — including the credential guard's coverage test, which derived both sides of its check from one filter, so restoring .log to the binary set skipped the assertion rather than failing it.

  • The strata table asserted a uniformity the artefacts contradict (#209). One row claimed all eight oracles share one stratified mixture. build_config.reference_sets.activity_derivation in the shipped NPZs records otherwise: LegNet drops gene_body, and ChromBPNet and Cherimoya drop gene_body and add a 5,000-position dhs stratum sampled from the DHS vocabulary. Split into three rows with the reason for each deviation, and pinned to the provenance rather than to memory. This table is the methods description behind every percentile chorus publishes, so a mixture it misstates is a mixture readers will cite.

  • Two documents disagreed about whether the history was rewritten (#207, #211). audits/README.md explained that dropping the audit artefacts would not shrink git clone and closed "Making the clone smaller would require rewriting history … deliberately not done" — while this file opens the release with instructions to re-clone because it was done. A reader arriving from audits/ hit the wrong one. Corrected, and the same file's branch-cleanup note fixed: release/v0.7.2 is #175 and was already inside the #169–#177 range, so 18 deleted branches had been enumerated as 19. The #198 entry above also gave audits/ as 119 MB while stating its 347 artefacts alone were 122 MB — a subset larger than its total, now ~123 MB.

  • The audit runbook still asked for the artefacts this release removed (#211). audits/AUDIT_CHECKLIST.md — the file CLAUDE.md names as the thing to run before any release — listed screenshots/*.png, nb_fresh_output/*.ipynb and two probe logs under "a full audit should leave behind, in audits/…". Followed literally, the next audit re-commits the 347 files and ~122 MB that #198 removed. It now says to produce them, quote what matters into report.md, and keep them outside the repository.

  • Regenerating the walkthrough notebooks no longer rewrites all 13 of them for no reason.
    nbformat.v4.new_code_cell mints a random id per call, so every run of
    scripts/generate_walkthrough_notebooks.py produced 121 insertions and 121 deletions of which
    zero lines were anything but "id". The churn was not the problem; the problem was that
    "have the committed notebooks drifted from their generator?" became unanswerable, because
    git status was dirty after every run whether anything had changed or not — so a real drift would
    have hidden in noise nobody reads. Cell ids are now derived from (index, cell_type, source):
    stable across runs, unique within a notebook, and changed exactly when a cell's content changes,
    so a one-cell edit produces a one-cell diff.

  • Three of the six library notebooks pointed at the wrong Jupyter kernel
    (#220).
    cherimoya_quickstart.ipynb,
    epinformerseq_testing.ipynb and klf1_validated_enhancer_profiles.ipynb declared
    kernelspec.name = "python3", which resolves to whatever the reader's default Jupyter kernel is.
    On the dev host that accidentally is the chorus env, so the problem was invisible; anyone starting
    JupyterLab from a base or system Python gets their own interpreter and fails on the first
    import chorus. klf1 was the worst of the three — display_name: "chorus" with
    name: "python3", so JupyterLab showed the reader the word "chorus" while running something else,
    and a wrong kernel that announces itself as the right one is harder to diagnose than a missing one.
    All six now declare the kernel chorus setup registers; metadata only, with every output cell
    preserved.

  • examples/walkthroughs/README.md promised more than it shipped
    (#220).
    Its title reads "pre-run, MCP-driven
    worked examples", which is true of the reports — every directory ships generated HTML, JSON and TSV
    — and not true of the notebook.ipynb beside them, all 12 of which carry zero outputs because
    they are code-generated and executing them needs the matching oracle env and usually a GPU. A reader
    who opened one expecting the promised results found an empty file. Now stated directly under that
    title, pointing at examples/notebooks/ for notebooks that do ship with their results.

Changed

  • audits/ no longer ships its raw artefacts, only the reports
    (#198).
    It was 426 of 869 tracked files —
    nearly 3× the package itself — and ~123 MB, which made the repo's file tree read as an audit dump
    rather than a library. The split was lopsided enough to decide it: 79 .md reports hold all the
    reasoning in 0.9 MB
    , against 347 artefacts in 122 MB. The reports stay, and several are cited
    from this file and from CLAUDE.md; the artefacts were evidence for conclusions the reports
    already state. A full copy was taken first and audits/README.md records that it exists, so nobody
    assumes the evidence is gone. Tracked files 869 → 524.

    Also found there: this file cited
    audits/2026-04-26_v29_scorched_earth/probes/05_html_render.py, which never existed —
    git log --all returns nothing for that path and it is absent from the pre-removal archive. That
    citation was broken before the cleanup, not by it, and now says so.

  • resume.md is no longer in the repo root
    (#197).
    18 KB of internal working notes titled
    "Resume notes — chorus, updated 2026-08-03 (rebuild in flight)", opening with a table of "Claims
    that were made and then REFUTED" and hardcoding /home/nvidia paths. Root-level files are the
    second thing a newcomer clicks, and "rebuild in flight" eleven days before the tag reads as
    mid-surgery. Removed from the tree; git history retains it.

  • There is one procedure for adding an oracle now, and it is CONTRIBUTING.md
    (#189).
    Three documents described it and they
    disagreed on the central fact: docs/IMPLEMENTATION_GUIDE.md told contributors to register in an
    ORACLE_REGISTRY that has never existed in this codebase — following it produces a dict
    nothing reads and a ValueError: Unknown oracle at run time — while environments/README.md said
    the environment "will be automatically detected by the CLI", true of chorus list and false of
    everything else. CONTRIBUTING.md was the only one correct as far as it went, but never mentioned
    ORACLE_SPECS, the dependency probes, training_genome, or background nulls, so following it
    end-to-end still produced an oracle invisible to MCP that returned None for every percentile.

    Step 5 now lists the sites that actually exist — each verified present before being written down,
    since documenting a site that isn't there is the defect being fixed — and the other two shrink to
    cross-references. tests/test_registries_cover_every_oracle.py checks them mechanically.

    One registry was deleted rather than documented: ORACLE_CLASS_MAP had no live consumer. Its
    only caller assigned the lookup to a local and never read it, because the generated script asks the
    child process for its own __class__.__name__. It had gone stale unnoticed (no alphagenome_pt,
    and its fallback would have produced Alphagenome_ptOracle against the real
    AlphaGenomePTOracle). Completing it, or guarding its completeness, would have enshrined an
    invariant nothing relies on.

    AUDIT_PROMPT.md is removed. Nothing referenced it, it duplicated
    audits/AUDIT_CHECKLIST.md — which CLAUDE.md names canonical — and it carried 24 stale claims
    including git checkout chorus-applications, "six oracle environments", "286 passed" against 1,960
    collected, and four builder flags that do not exist. Its two genuinely unique pieces are folded into
    the checklist as §8b/§8c; its Selenium block is retired in favour of the playwright suite added
    earlier in this release, which pierces shadow roots.

  • docs/BACKGROUND_NULL_PROTOCOL.md is current with the count-head extraction (#186). The protocol is the living document a new oracle's background build is written against, and §8 said nothing about which quantity a CDF must be built from — the distinction #125 exists to enforce. It now carries §8 Step 2b (use chorus.core.count_head.expected_counts_profile from both the oracle and the builder, because a CDF is only meaningful if it was built from the quantity predict() returns), the three count-head conventions side by side (log1p per track → expm1; log1p pooled → exp(C) − n_tracks; log10 → 10**C, which differ by 26× at log-count 2.5), a §7 equivalence-guard row, and a §9 decision-log entry.

  • CI renders reports on every PR now, a reduced set of them. The full 19-report browser suite stays on the release host, but running none of it in CI is how a blank panel reaches main between audits — which is the gap that let a size ceiling stand in for a rendering check in the first place. CHORUS_BROWSER_SMOKE=1 selects the two smallest IGV reports plus the panel-less table: 12 tests instead of 46, enough to catch the failures that hit every report at once.

    This needed test_ci_runs_the_same_command_a_contributor_runs narrowed, since it correctly refused the new job. Its property is about the run that stands for "the tests pass" — a whole-suite invocation must carry no selection of its own — so a single-file job with an explicit marker is now allowed while a widened one still fails. Mutation-tested three ways.

  • Six items in audits/AUDIT_CHECKLIST.md corrected, because they produced false findings against a correct tree: the <details> count (the multi-oracle report legitimately has four), the CDN grep (it matches googleapis and igv.org inside the inlined igv.js, firing on all 19 reports — the measured request inventory replaces it), the P0 forbidding the pre-rename examples directory (its three live mentions are the test and the comment that record the rename, not paths anyone is sent to), list_tracks being an MCP tool rather than a Python API, the dependency-pin check, and which function §14's padding item refers to.

  • ⚠️ AlphaGenomeOracle(organism="mouse") now raises instead of being silently ignored (#124). The parameter was accepted, assigned to self.organism, and read by nothing — the metadata loader hardcodes Organism.HOMO_SAPIENS and the PyTorch port passes organism_index=0 — so the one oracle whose upstream API genuinely supports mouse had a switch that looked functional and returned human predictions under a mouse label. Same fix on AlphaGenomePTOracle. organism="human" (any capitalisation, plus homo_sapiens) is unchanged; anything else raises NotImplementedError naming what mouse would actually require. Of make-it-work / remove-it / raise, only raising is both honest and affordable: mouse needs an mm10 reference in the genome manager, an mm10 reference class for the background null (SCREEN publishes mm10 cCREs; the Meuleman DHS index has no mouse equivalent), and a background pass over ~4,300 further tracks.

  • Report size is no longer a proxy for report health. The ceiling stands at 50 MiB — GitHub's advisory threshold, under its hard 100 MiB wall — and it now exists alongside a check that actually loads the file. Measured across the corpus, size barely predicts load time: 20× the bytes costs 1.3× the wait (25.7 MiB → 11.3 s; 1.3 MiB → 8.8 s), because most of those seconds go on ~14 network round-trips for genome resources (#139) rather than on parsing the payload. CHORUS_WRITE_LARGE_HTML is gone.

  • The BPNet-family count-head arithmetic now has one implementation (#125). Turning a profile head and a count head into a per-position expected-count track is four operations — centre the logits, softmax, invert the count head, scale — and it existed in five places at once. All three defects fixed on 2026-07-31 were two of those copies disagreeing: exp vs expm1 across four call sites (+1 read, ~0.1% at a peak but up to 100% at a quiet site, which is the regime the activity CDFs are built from); per-strand vs joint softmax (the two emitted tracks together claimed 2.00× the predicted counts); and a count bias hardcoded (N, 1) that Keras broadcast silently (every log-count shifted by 0.5885, i.e. 1.80× low at a peak and 3.04× at a quiet site). None crashed; each produced a plausible number and each shipped.

    chorus/core/count_head.py is now the only implementation, used by ChromBPNetOracle's three paths, Cherimoya's scoring.py, and the ChromBPNet background builder — the last of which had to be factored into a profiles_from_heads() function, because its arithmetic was inline in a TensorFlow-dependent routine and could therefore only ever be compared against the oracle by grepping both files for matching source text. That is exactly the check exp/expm1 walked past four times.

    No number moved, and that was verified rather than assumed. The pre-extraction expressions are written out verbatim in the tests and compared with array_equal, not allclose, in both float32 and float64. End to end, regenerating the ChromBPNet and Cherimoya examples produced JSON and HTML differing from the committed files only in the timestamp, so no example needed re-committing and the shipped CDFs are untouched.

    The one thing that did drift was caught by that end-to-end run and by no unit test: the two copies disagreed about precision — Cherimoya cast to float64 before doing anything, ChromBPNet used whatever TensorFlow returned. The shared helper therefore preserves its caller's dtype, and Cherimoya promotes on the way in; without that, leaving log_counts in float32 moved the SORT1 example's ref_value from 603.3464052301788 to 603.3464123072064 — 1.2e-8, and with nothing behind it. Whether ChromBPNet should compute this in float64 is a real question and deliberately a separate one.

    Two copies stay where they are, on purpose, with their equivalence pinned. The torch expression in build_backgrounds_cherimoya.py runs on the accelerator inside the batch loop, where a numpy round-trip per batch would be charged to a multi-hour job — verified in chorus-cherimoya (torch 2.13.0+cu130) at a 1.25e-07 maximum relative difference. And EPInformer-seq is not routed through here at all: it scales by 10 ** log_count, a different convention from a differently-trained model, and at a log-count of 2.5 the two differ by 26× — so a tidy-up that unified them would be the same class of mistake as the three above, in the opposite direction. A test pins that difference rather than leaving it to be noticed.

  • The browser job no longer fails when UCSC does not answer (#206, #210). All 19 committed reports resolve their reference sequence from hgdownload.soe.ucsc.edu, because igv.js requires a sequence source and hg38 is ~3 GB — a documented limitation. The unintended consequence was that the rendering suite's verdict depended on a third party: when UCSC refused a connection during one PR's run, both smoke reports reported canvases 0/0 painted and blew the 30 s budget at the 60 s timeout — four failures, none about the reports, on a change that touched only CONTRIBUTING.md. A suite that reddens when someone else's server hiccups teaches people to re-run CI until it is green, which is how a real blank-panel regression gets waved through.

    The first version of that tolerance was too generous in three ways, all found by review: it swallowed uncaught JS (the skip ran before the page_errors assertion), it accepted any failing external URL rather than the reference host — satisfied essentially always, since every report contacts UCSC — and it would have skipped a wrong URL committed inside a report forever, that being a permanent defect rather than weather. A first attempt to fix the last one compared with host in url, which also matches hgdownload.soe.ucsc.edu.example.net; it is now an exact netloc comparison. One case stays undecidable and is documented as such: a correct reference URL that is permanently dead is indistinguishable from an outage in a single render.

  • pytest.ini no longer claims an exact integration-test count (#205). The usage comment named a figure that had drifted four times (66, 153, 154, 157). Pinning it by collection failed on CI, which collects 125 where the dev host collects 162 — because integration tests parametrize over locally-present artefacts, so the count is a property of the machine and never was a fact about the repository. The comment now gives a magnitude with the reason, and the guard asserts what is checkable: the marker must still select at least 100 tests, so "the marker stopped being applied" and "a collection error is swallowing the suite" still fail.

  • CLAUDE.md keeps its copy of the regeneration matrix, now checked (#208). #203 moved that matrix into CONTRIBUTING.md precisely because two copies drift, then left the original in place with only the new copy guarded. Both are now parametrized over the same argparse-derived assertions — pinned to the code rather than to each other, since two copies compared with one another can agree and both be wrong.

  • The install instructions pin a released tag. git clone followed by pip install -e . installed
    whatever main happened to be, while the README's numbers, the committed example outputs and the
    background CDFs are only consistent as of a tag. The quickstart now checks out v0.7.3 and says
    when to omit it. Also states plainly that chorus is not on PyPI — the name chorus belongs to an
    unrelated chemistry toolkit — so pip install chorus fetches someone else's package.

Added

  • The installation instructions were audited against the code and re-measured; the disk figure was
    wrong by more than 2× and is the reason to re-read them.
    The prerequisite said ~38 GB free
    disk
    while a default all-oracle install on Linux + CUDA measures ~85 GB — so anyone who
    provisioned a 40 GB volume on that advice ran out of space partway through an unattended
    55–75-minute setup, with a half-built set of envs and no hint that 2× was needed. Two independent
    causes: the per-env row claimed "~3 GB each" when the smallest oracle env is 6.1 GB and the largest
    is 11 GB (every env carries its own CUDA payload — pip nvidia_* wheels in six of them, conda-side
    libtorch_cuda/libcu* in Borzoi, LegNet and Sei, and pip does not hardlink between envs); and the
    table had no row at all for the weights chorus setup prefetches by default, ~11 GB of which
    Sei alone is 6.5 GB because its 3.1 GB tarball is kept beside the 3.4 GB it extracts to. The
    breakdown is now itemised to 14 buckets and tests/test_disk_claims_add_up.py fails if the rows
    stop summing to the stated total or the prerequisite drops below it.

    Fixed in the same pass, each verified against the code rather than re-read:

    • chorus setup all is not a command — the subparser defines only flags, so it exits 2 with
      "unrecognized arguments". It was the documented prerequisite in
      examples/walkthroughs/README.md, and appeared in chorus/utils/ld.py's user-facing hint and in
      _setup_all.py's own failure banner.
    • chorus setup needs a TTY. The token resolves before anything is built and aborts if stdin
      is not a TTY, so nohup chorus setup & — the pattern the TLDR recommends — died instantly with
      zero progress. HF_TOKEN / --hf-token / --no-weights are now documented there.
    • Notebooks needed a step that did not exist in the docs. 16 of 19 shipped notebooks declare
      kernel name chorus, nothing in the package registers it (grep -rn ipykernel chorus/ → 0), and
      the README claimed all three "work as soon as chorus setup finishes". Without
      python -m ipykernel install --user --name chorus, nbconvert raises NoSuchKernel.
    • claude mcp add chorus … does not register globally — it defaults to --scope local, so the
      documented "available in every project" recipe registered chorus for the clone directory only.
      Now -s user.
    • chorus cleanup --all leaves the HuggingFace cache, which is where most weights live (~20 GB),
      while the docs called it "Remove everything".
    • ~/.chorus/backgrounds/ was wrong in every live place it appeared — 32 lines across 30 files, including two CLI help strings and four
      normalization.py docstrings stating it as the default. The data directory has defaulted to the
      installation tree since 2026-08; CHORUS_DATA_DIR — the one switch that relocates all 85 GB —
      appeared in no user-facing doc at all, and now has its own section, including the legacy
      backgrounds-only rule that lets backgrounds resolve outside data_dir.
    • $(chorus config data-dir) was used as a path in this file's own air-gap recipe; the command
      prints a human-readable block.
    • Stale by 28×: three _setup_prefetch.py comments called the 2-model ChromBPNet default
      "~1.4 GB" where the slim mirror serves 25 MB each. The giveaway was that the same line put the
      786-model catalogue at ~1.5 GB.
    • environments/chorus-base.yml is documented as vestigial — no code reads it (the manager
      explicitly excludes it), it is not the subset the docs claimed, and it declares name: chorus,
      so installing it collides with the documented base env.
    • Three dead in-page doc anchors, now guarded by tests/test_doc_links_resolve.py.
  • The genome assembly is now asserted rather than assumed (#124). Chorus is human hg38 everywhere, and until now that held by accident: Enformer and Borzoi exclude their 1,643 and 2,608 mouse tracks because someone selected enformer_human_targets.txt / borzoi_human_targets.txt, AlphaGenome because Organism.HOMO_SAPIENS is hardcoded in its metadata loader. Those are file and literal choices, not assertions — nothing connected any of them to genomes/hg38.fa, which every builder opens, so nothing would have caught a future *_mouse_targets.txt. ChromBPNet, whose registry had no organism field at all, is where it actually went wrong: 33 mm10 models were scored against hg38 sequence using the hg38 DHS vocabulary, removed in #121.

    A hard failure rather than a warning, because there is no symptom to notice: mm10 chr1:1,000,000 exists in hg38 too, so every coordinate resolves, every prediction returns, and every percentile lands in [0, 1]. The answer is simply about a different piece of DNA. Four mechanisms:

    • every oracle declares training_genome on its own class — deliberately not inherited, since OracleBase defaulting to "hg38" would be one more silent choice, and a test enumerates the subclasses to make sure a new oracle says;
    • all 8 builders check it against the FASTA they open, at all 10 open sites, via require_reference_assembly — at preflight, so a 14-hour build fails in its first second;
    • both stamp scripts observe the assembly from the reference's chromosome lengths instead of writing "genome": "hg38" as a literal, which had made that field a restatement of the stamper's own assumption;
    • the loader refuses an artefact declaring a genome chorus does not rank against, as BackgroundGenomeMismatch.

    Identity comes from chr1's length, which is provider-independent: UCSC, Ensembl and GENCODE disagree about chromosome naming, line width and scaffolds but agree on chromosome lengths, where the fasta_sha256_prefix64mb the artefacts already carry would reject a correct Ensembl GRCh38 as loudly as it rejects mm10. An unrecognised reference warns and proceeds — refusing it would break anyone on a legitimate custom build to enforce a lookup table's completeness — while a recognised, wrong one raises. A typo in the expected value ("GRCh38" for "hg38") also raises, so it cannot silently disable the check it was added to perform.

    No published number changes and no artefact was re-uploaded. All nine shipped backgrounds already declared genome: hg38, and all nine are; the fix is that this is now checked at both ends rather than asserted at neither. Deliberately not done: the per-row genome field the issue also proposed, which would mean re-stamping and re-uploading all nine artefacts and re-cutting the pinned dataset tag, and only earns its keep if chorus ever ships mixed-species artefacts.

  • BackgroundGenomeMismatch and BackgroundFoldMismatch now share a BackgroundArtefactMismatch base, and get_normalizer re-raises the base class. Every other reason a per-track artefact fails to load means "no percentiles available", which its legacy .npy fallback is right to absorb; these mean "the percentiles would be wrong", ranked against a different distribution than they name. Catching the family rather than listing subclasses means the next guard of this kind inherits the non-swallowing contract instead of depending on someone remembering to widen the clause.

  • Every committed report is now opened in a real browser and checked that it paints (#135). Nothing in the suite had ever loaded one: the checks were on bytes and JSON, so a report could be regenerated, sized, diffed, committed and shipped while rendering a blank page. tests/test_committed_reports_render_in_a_browser.py loads all 19 in headless Chromium and asserts every canvas IGV laid out has ink, with no console errors and no uncaught exceptions. Marked integration; skips cleanly where Chromium is unavailable.

    Three things had to be got right, each of which produces a failure indistinguishable from a broken panel. IGV renders inside a shadow root, so document.querySelectorAll('canvas') returns 0 for a panel that is painting perfectly — which is what led #139 to report "the panel never renders", and to the conclusion that headless Chromium cannot be used as an oracle for this. It can; the query just has to pierce shadow boundaries. Chromium's shared libraries are scattered across conda envs (libgbm/libatk only in chorus-browsertest, libXcomposite/libcups only in the oracle and base envs), so the path is assembled in the harness and passed to the browser process rather than exported by whoever runs pytest. And "loaded" is not "painted": polling until the canvas count stops changing reported 61/62 and 39/42 on reports that reach 64/64 and 44/44 a second later, so the harness polls to convergence instead.

    Calibrated rather than guessed: "painted" is any non-white pixel, because a 0.05% floor was too high — the causal report's point tracks are ~20 marks across a 3288 px canvas (0.000238 of it) and were being reported as blank while drawing exactly what they should. And the check is shown to work: three mutations of a real report (features emptied, genome name broken, config truncated) are each caught by a different one of the three channels.

  • The CDYL fine-map report ships its IGV panel — examples/walkthroughs/causal_prioritization/CDYL_rs9504151/rs9504151_CDYL_locus_causal_report.html, 25.7 MiB and now the largest artefact in the repo. It was the one example with no browser panel, skipped behind CHORUS_WRITE_LARGE_HTML=1, because 21 lung-fibroblast tracks × 2 alleles exceeded a 20 MiB ceiling that had been chosen as "headroom over today's largest artefact" rather than derived from any failure. Adding it moved no numbers: regenerating the example produced a JSON differing from the committed one in 1 of 4,001 leaf values, and that one is the timestamp.

  • Contributing a worked example is a documented path (#203). CONTRIBUTING.md opened by describing itself as a guide to implementing a new oracle, and Current Priorities listed three kinds of contribution, all models — so a reader with a variant they cared about and no intention of porting a network was turned away in sentence one. The mechanics were undocumented too: four regeneration scripts own different example types, and the matrix mapping example → script → valid --oracle → conda env lived in CLAUDE.md, this changelog, and a May audit report. It exists because getting it wrong is the most common way a regeneration silently does nothing — regenerate_multioracle.py accepts chrombpnet/cherimoya/legnet/alphagenome and pointedly not enformer, has no --gpu where the other two do, and the wrong conda env logs Failed to load <track> per track and carries on rather than failing fast. There is now a routing table in the first screen (oracle / example / small fix, with honest sizes), a section carrying the verified matrix with the real entry shape and the four traps, and the six guard tests to expect — whose documented command collects 138 tests. Eight new tests pin the table to the scripts' argparse definitions, including the no-enformer exception in both directions, so the table cannot go stale silently.

  • An outside contributor can develop an oracle without our HuggingFace dataset (#201). The capability existed and was documented nowhere: CHORUS_BACKGROUNDS_REPO redirects the backgrounds lookup to a repo you control, and get_pertrack_normalizer(name, cache_dir=…) is checked before any download, making the path fully offline — yet neither appeared in README, CONTRIBUTING, BACKGROUND_NULL_PROTOCOL.md or NORMALIZATION_GUIDE.md. Read as written, §8 said percentiles require a per-track null, the canonical nulls live in a dataset you cannot write to, and building one means scoring ~18,000 positions through your model — from which a contributor reasonably concludes they need the lab's infrastructure and stops. Both mechanisms are now documented and pinned by tests, and CONTRIBUTING states plainly that a small background, or none with percentiles labelled unavailable, is still a useful PR.

  • chorus setup registers the chorus Jupyter kernel. 15 of the 18 shipped notebooks declared
    kernelspec.name == "chorus" and nothing created it, so on a fresh install every one of them failed:
    jupyter nbconvert --execute raises NoSuchKernel: No such kernel named chorus, and JupyterLab
    silently prompts for a kernel instead — which reads as a broken notebook rather than a missed step.
    The step existed only as item 4 of examples/notebooks/README.md, a file you reach after deciding
    to open a notebook. Opt out with --no-jupyter-kernel.

    Registration uses --user, not --sys-prefix, so the kernel is visible to a JupyterLab started from
    any environment — with --sys-prefix the notebooks still fail from a system Jupyter, which is how
    most people run one. Failure is never fatal: a missing ipykernel or an unwritable Jupyter config
    logs a warning and the exact command to run later rather than turning a completed multi-GB install
    into a non-zero exit.

  • CHORUS_IGV_BUNDLE_SEQUENCE=1 makes an HTML report render with no network at all. Every report
    resolved its reference sequence from hgdownload.soe.ucsc.edu, so opened offline the IGV panel
    painted nothing — measured canvases 0/0, because igv.js allocates no canvases if it cannot
    build a genome, and simply dropping the sequence is worse: with neither fastaURL nor twoBitURL
    it falls back to resolving id: "hg38" against igv.org's hosted registry, the remote catalogue
    fetch #139 removed.

    The way through is that igv.js takes chromosome lengths from chromSizesURL, which reports already
    inline, and needs the FASTA only to initialise. So a 1 kb placeholder is enough: the same report
    went from 0/0 painted in 45.3 s (timeout) to 100/100 painted in 1.6 s with every external
    request blocked.

    Opt-in, and the placeholder is deliberately N rather than real bases. An unindexed FASTA is
    positioned from offset 0 of its contig while these reports display loci tens of megabases in, so
    real sequence would render at the wrong coordinates — worse than rendering none, because a reader
    can zoom in and read it. CHORUS_IGV_SEQUENCE_URL still serves the real thing for anyone who wants
    both, and remains the default path.

Known limitations

  • The SORT1_enformer example cannot be regenerated reproducibly. Two runs of identical code
    differ in 596 numeric fields at a 0.0159% median (ref_value max 0.067%, raw_score max
    4.29% where log2FC amplifies a near-zero denominator). That is ~298 distinct values, not 596:
    each track is serialised twice, once under all_scores and once under scores_by_layer. Of the
    file's 845 numeric leaves, that is most of them. Enformer's forward pass is bitwise identical
    in-process and across processes, so the cause is above the model and is still not localised.
    Measured in audits/2026-08-12_post_v0.7.2_audit.md (F8); narrowed further in
    audits/2026-08-14_f8_localisation.md.

    It is intermittent, which is why it looked contradictory. Four pairs of consecutive
    use_environment=True predict() calls over the same region: three pairs bitwise identical, one
    pair differing in 3,583 of 3,584 values. That reconciles "the forward pass is bitwise identical
    across processes" with "regenerations keep differing" — both are true, and a single clean probe is
    not evidence of determinism at this failure rate. Anyone chasing it further must run each
    configuration several times and report a rate; a one-shot comparison confirms whichever answer
    the experimenter expects about three times in four.

    Two things about it are now pinned rather than assumed. The top-N cutoff is a discontinuous
    amplifier:
    discover_variant_effects cuts hard at top_n_per_layer, and the gap at
    tf_binding's cutoff in the committed example is 4.25% — smaller than the 4.29% drift
    measured on raw_score — so a wobble that size can change which track ships, presenting as a
    large diff rather than a small one. The percentile tie-break is not the mechanism: its
    blake2b draw is keyed on the raw value, so a 1-ULP change could in principle re-hash across a
    whole tied run, but measured on all 84 committed tracks, 0 change rank under
    np.nextafter.

    scripts/gate_end_to_end_determinism.py does not catch it — but not because "what it compares is
    deterministic", as this entry previously said. The gate is cross-process (it spawns two
    children); it is pointed at AlphaGenome with use_environment=False and compares numeric
    leaves only. It now takes --oracle enformer and reports string changes too, so a top-N
    identity flip is named as such instead of masquerading as numeric drift.

  • A report still fetches one thing from the internet: the reference sequence. Every igv.js
    version requires a sequence source and hg38 is 3 GB, so it cannot be bundled. Everything else is
    inline. Point CHORUS_IGV_SEQUENCE_URL at a self-hosted copy served same-origin with the
    report and it needs no network at all (measured: 0 requests, 0.8 s).

  • Cherimoya and ChromBPNet remain incomparable at raw magnitude — CATv1 has no bias model and
    tracks bias-aware ChromBPNet, while chorus loads chrombpnet_nobias. Percentiles are
    unaffected, since each is ranked against its own null. Unchanged from 0.7.2.