Skip to content

Read every tutorial CSV with float_precision="round_trip" - #94

Merged
shanikawm merged 1 commit into
mainfrom
fix/csv-float-precision
Aug 7, 2026
Merged

Read every tutorial CSV with float_precision="round_trip"#94
shanikawm merged 1 commit into
mainfrom
fix/csv-float-precision

Conversation

@shanikawm

Copy link
Copy Markdown
Contributor

Follow-up to #93, which fixed this in the DE tutorial only and left 42 read
sites across 17 others.

pandas' default CSV reader is not correctly rounded — it misparses about a
third of random doubles by an ULP. to_csv was never at fault; it already
writes the shortest round-trippable form. So a tutorial that wrote a value,
handed it to R, read both back and reported "these agree exactly" was partly
measuring the parser rather than the port.

What changed

37 call sites across 16 tutorials. Three reads stay exempt with stated
reasons: cell labels in anchors_tutorial, a row count wrapped in len() in
visium_tutorial, and the DE hex reader, which parses floats via
float.fromhex.

Six reported figures moved, all at the ULP level and most of them downward:

Tutorial Figure Before → After
PBMC 8k percent.mt 5.773e-15 → 5.329e-15
PBMC 3k VST mean, max rel 1.548e-14 → 4.973e-15
PBMC 3k VST variance, max|diff| 1.592e-11 → 1.614e-11
PBMC 3k marker row 7→8 4.80e-14 → 4.62e-14
SCTransform detection_rate 5.6e-16 → 5.0e-16
CITE-seq min 2.082e-16 → 2.498e-16

No declared band moved. All nine --report runs still exit zero, and
re-running them reproduces the new values exactly. The four affected vignettes
carry the measured numbers. The [0.9.0] changelog entries quoting the old
figures are left alone — they were accurate at that release, and a changelog is
a record rather than a live table.

A near miss worth recording

The first attempt inserted the argument before the last ) on the line. For
pd.read_csv(path).set_index("cell") that lands it inside set_index
syntactically valid, passes an AST check, and would have raised TypeError at
runtime on 16 of the 37 sites. Caught only by reading back every changed line
rather than trusting the parse. Redone with paren matching from the
read_csv( position.

Guard

tests/test_tutorial_csv_precision.py pins the convention across all eighteen
tutorials so the next pd.read_csv cannot quietly reintroduce it. It also
asserts the three exemptions still match something — a stale exemption stops
matching and silently permits a real offender in the same file. Both guards
mutation-tested: stripping the parameter from one call fails that file's case,
and editing an exempt call's marker fails both the file case and the
exemption check.

Also corrected

PBMC 3k's docs claimed avg_log2FC agreement "to 4.9e-15" for both clusters
whose cells match exactly. That covers the 151-gene cluster (4.88e-15); the
242-gene one is 4.62e-14, ten times larger. Both figures are now stated.
This predates the reader fix — the bound was simply the smaller of the two.

Verification

1168 passed, 25 skipped. mypy at its 4-error baseline. No figure churn.

Not in this PR

The R half is not lintable from Python and is unchanged: write.csv renders 15
significant digits, and raising it does not help because R's own
sprintf("%.17g") is not correctly rounded either. Only the DE tutorial has the
C99 hex-float (%a) side table. The tutorials making the strongest exactness
claims — the object model ("91 of 91 anchors, no tolerance"), Visium
(max|Δx| = 0), out-of-core ("bit-identical"), and spatial (Moran's I to
1.6e-14) — still cross the language boundary through plain CSVs.

🤖 Generated with Claude Code

pandas' default CSV reader is not correctly rounded -- it misparses about a
third of random doubles by an ULP. `to_csv` was never the problem; it already
writes the shortest round-trippable form. So a tutorial that wrote a value,
handed it to R, read both back and reported "these agree exactly" was partly
measuring the parser rather than the port.

37 call sites across 16 tutorials fixed. Three reads stay exempt with stated
reasons: cell labels in anchors_tutorial, a row count wrapped in len() in
visium_tutorial, and the DE hex reader, which parses floats via float.fromhex.

Six reported figures moved, all at the ULP level and most downward: PBMC 8k's
percent.mt 5.773e-15 -> 5.329e-15, PBMC 3k's VST mean relative difference
1.548e-14 -> 4.973e-15 and its variance max|diff| 1.592e-11 -> 1.614e-11,
SCTransform's detection_rate 5.6e-16 -> 5.0e-16, CITE-seq's minimum 2.082e-16
-> 2.498e-16, and one PBMC 3k marker row 4.80e-14 -> 4.62e-14. No declared band
moved; all nine --report runs still exit zero, and re-running them reproduces
the new values exactly. The four affected vignettes carry the measured numbers.
The 0.9.0 changelog entries quoting the old figures are left alone -- they were
accurate at that release and a changelog is a record, not a live table.

The first attempt at this patch inserted the argument before the last ")" on
the line, which for `pd.read_csv(path).set_index("cell")` lands it inside
set_index. That is syntactically valid, passes an AST check, and would have
raised TypeError at runtime on 16 sites. Redone with paren matching from the
read_csv( position, and every changed line read back.

tests/test_tutorial_csv_precision.py pins the convention across all eighteen
tutorials so the next pd.read_csv cannot quietly reintroduce it. It also
asserts the three exemptions still match something -- a stale exemption stops
matching and silently permits a real offender in the same file. Both guards
mutation-tested.

Also corrects a bound the docs got wrong independently of this: PBMC 3k claimed
avg_log2FC agreement "to 4.9e-15" for both clusters whose cells match exactly.
That is the 151-gene cluster; the 242-gene one is 4.62e-14, ten times larger.
Both figures are now stated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shanikawm
shanikawm merged commit c6ff32f into main Aug 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant