Skip to content

Serialize R anchors at full float64 precision - #95

Merged
shanikawm merged 1 commit into
mainfrom
fix/r-hex-handoff
Aug 7, 2026
Merged

Serialize R anchors at full float64 precision#95
shanikawm merged 1 commit into
mainfrom
fix/r-hex-handoff

Conversation

@shanikawm

Copy link
Copy Markdown
Contributor

Completes the precision work from #94, which fixed the Python side of the
R↔Python numeric handoff and left the R side.

Scope note up front: this changes no result. Every anchor matched before and
matches now. It is a correctness fix to the measuring instrument, and the
defect it removes was below the tolerance of every comparison that used it. See
Was this load-bearing? below for the numbers — I would rather state that than
let a green diff imply a save.

What was wrong

Four verify scripts wrote their JSON references at digits = NA, 12 or 15.
All three truncate. Measured against R 4.x on a 2,000-value spread of realistic
magnitudes:

digits round-trips float64?
NA no — loses ~half
12 no — loses ~half
15 no — loses ~12%
17 yes
22 yes

All four now use 22, matching visium_verify.R, which had already worked this
out and was the only one already correct.

Two counter-intuitive properties, both assumed wrong somewhere in this repo

  • digits counts decimal places, not significant digits. Whether a setting
    is lossy therefore depends on the magnitude of the value — which is why
    digits = 12 looked fine on anchors in the thousands while silently
    truncating the ones near zero.
  • NA is not "max precision". It round-trips fewer doubles than an explicit
    17. lazy_bpcells_verify.R carried a comment recommending NA for exactly
    the reason that makes NA wrong — the diagnosis was right and the fix was
    not. That comment is corrected rather than deleted.

Note this is the opposite of the R CSV finding in #94, where raising the
digit count does not help because sprintf is not correctly rounded. jsonlite
has its own serializer and is correctly rounded, so no hex-float side table is
needed here — one parameter per script suffices.

What moved

Reference Values that gained precision Example
out-of-core 203 of 309 1.635873317718511.6358733177185059
Xenium spatial 37 of 60 24.07388412120224.073884121202223
Xenium SVF 4 of 81 Moran's I, 16 → 17 significant figures
object model 0 of 160

The object-model script was losing nothing: every float in it is rounded to 6 dp
on both sides before serialization, so 15 significant digits was ample. Changed
anyway, because NA is a trap for the next anchor added without rounding.

Was this load-bearing?

No, and the margins say so:

Worst truncation error Tightest tolerance Margin
Xenium SVF 1.5e-16 1e-9 6,500,000×
Xenium spatial 4.4e-13 1e-6 2,280,000×
out-of-core 4.2e-15 1e-13 24×

The value is preventative. If anyone tightens mem.normalize_head from 1e-13
toward exactness — the natural direction for a fidelity project — it would begin
failing for a serialization reason rather than a numerical one, and that is a
hard failure to diagnose. Now it cannot.

One genuine reader-facing fix

The extra precision exposed a display bug in the out-of-core report: a hard
22-character slice rendered 1.0153110547861388e-06 as
1.0153110547861388e-0, dropping the exponent's last digit. A reader would take
that for 1e-0 — a six-order-of-magnitude misread of a number the vignette
presents as evidence. Values are now shown whole or elided with an ellipsis, and
a 1-element list from the R side is unwrapped so the number gets the column
instead of the brackets.

Verification

All four R scripts re-run; all four comparisons re-run afterwards.

  • object model: 91/91 anchors match, report byte-identical
  • Xenium SVF: report byte-identical
  • Xenium spatial: all deterministic anchors match
  • out-of-core: exits zero, values now displayed at full precision

1168 passed, 25 skipped. tutorials/figures_spatial/r_reference.json is the one
tracked R reference, so its regeneration is included in the diff.

🤖 Generated with Claude Code

Four verify scripts wrote their JSON references at digits = NA, 12 or 15. All
three truncate. Measured on a 2,000-value spread of realistic magnitudes: NA
and 12 lose about half the values, 15 loses ~12%, and 17 and 22 are exact. All
four now use 22, matching visium_verify.R, which had already worked this out.

Two properties of jsonlite's `digits` drove the confusion, and both were
assumed wrong somewhere in this repo:

  * It counts DECIMAL PLACES, not significant digits. Whether a setting is
    lossy therefore depends on the magnitude of the value, which is why
    digits = 12 looked fine on anchors in the thousands and silently truncated
    the ones near zero.
  * NA is not "max precision". It round-trips fewer doubles than an explicit
    17. lazy_bpcells_verify.R carried a comment recommending NA for exactly the
    reason that makes NA wrong; the diagnosis was right and the fix was not.
    That comment is corrected rather than deleted.

What moved, after re-running all four R scripts:

  out-of-core     203 of 309 values   1.63587331771851 -> 1.6358733177185059
  Xenium spatial   37 of  60 values   24.073884121202  -> 24.073884121202223
  Xenium SVF        4 of  81 values   Moran's I, 16 -> 17 significant figures
  object model      0 of 160 values   already rounded to 6 dp on both sides

The object-model script was losing nothing, because every float in it is
rounded to 6 dp before serialization. Changed anyway: NA is a trap for the next
anchor added without rounding, and closing it costs nothing.

No comparison outcome changed. 91/91 object-model anchors, all Xenium
deterministic anchors, and every out-of-core and SVF anchor still match -- now
against R's true values instead of truncated ones. This is a correctness fix to
the measuring instrument, not to a measurement.

Also fixes a display bug the extra precision exposed: the out-of-core report
sliced values at 22 characters, which rendered 1.0153110547861388e-06 as
1.0153110547861388e-0 and dropped the exponent's last digit -- a reader would
take 1e-06 for 1e-0. Values are now shown whole or elided with an ellipsis, and
a 1-element list from the R side is unwrapped so the number gets the column
rather than the brackets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shanikawm
shanikawm merged commit dea7ee4 into main Aug 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant