Serialize R anchors at full float64 precision - #95
Merged
Conversation
Four verify scripts wrote their JSON references at digits = NA, 12 or 15. All
three truncate. Measured on a 2,000-value spread of realistic magnitudes: NA
and 12 lose about half the values, 15 loses ~12%, and 17 and 22 are exact. All
four now use 22, matching visium_verify.R, which had already worked this out.
Two properties of jsonlite's `digits` drove the confusion, and both were
assumed wrong somewhere in this repo:
* It counts DECIMAL PLACES, not significant digits. Whether a setting is
lossy therefore depends on the magnitude of the value, which is why
digits = 12 looked fine on anchors in the thousands and silently truncated
the ones near zero.
* NA is not "max precision". It round-trips fewer doubles than an explicit
17. lazy_bpcells_verify.R carried a comment recommending NA for exactly the
reason that makes NA wrong; the diagnosis was right and the fix was not.
That comment is corrected rather than deleted.
What moved, after re-running all four R scripts:
out-of-core 203 of 309 values 1.63587331771851 -> 1.6358733177185059
Xenium spatial 37 of 60 values 24.073884121202 -> 24.073884121202223
Xenium SVF 4 of 81 values Moran's I, 16 -> 17 significant figures
object model 0 of 160 values already rounded to 6 dp on both sides
The object-model script was losing nothing, because every float in it is
rounded to 6 dp before serialization. Changed anyway: NA is a trap for the next
anchor added without rounding, and closing it costs nothing.
No comparison outcome changed. 91/91 object-model anchors, all Xenium
deterministic anchors, and every out-of-core and SVF anchor still match -- now
against R's true values instead of truncated ones. This is a correctness fix to
the measuring instrument, not to a measurement.
Also fixes a display bug the extra precision exposed: the out-of-core report
sliced values at 22 characters, which rendered 1.0153110547861388e-06 as
1.0153110547861388e-0 and dropped the exponent's last digit -- a reader would
take 1e-06 for 1e-0. Values are now shown whole or elided with an ellipsis, and
a 1-element list from the R side is unwrapped so the number gets the column
rather than the brackets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the precision work from #94, which fixed the Python side of the
R↔Python numeric handoff and left the R side.
Scope note up front: this changes no result. Every anchor matched before and
matches now. It is a correctness fix to the measuring instrument, and the
defect it removes was below the tolerance of every comparison that used it. See
Was this load-bearing? below for the numbers — I would rather state that than
let a green diff imply a save.
What was wrong
Four verify scripts wrote their JSON references at
digits = NA,12or15.All three truncate. Measured against R 4.x on a 2,000-value spread of realistic
magnitudes:
digitsNA12151722All four now use
22, matchingvisium_verify.R, which had already worked thisout and was the only one already correct.
Two counter-intuitive properties, both assumed wrong somewhere in this repo
digitscounts decimal places, not significant digits. Whether a settingis lossy therefore depends on the magnitude of the value — which is why
digits = 12looked fine on anchors in the thousands while silentlytruncating the ones near zero.
NAis not "max precision". It round-trips fewer doubles than an explicit17.lazy_bpcells_verify.Rcarried a comment recommendingNAfor exactlythe reason that makes
NAwrong — the diagnosis was right and the fix wasnot. That comment is corrected rather than deleted.
Note this is the opposite of the R CSV finding in #94, where raising the
digit count does not help because
sprintfis not correctly rounded. jsonlitehas its own serializer and is correctly rounded, so no hex-float side table is
needed here — one parameter per script suffices.
What moved
1.63587331771851→1.635873317718505924.073884121202→24.073884121202223The object-model script was losing nothing: every float in it is rounded to 6 dp
on both sides before serialization, so 15 significant digits was ample. Changed
anyway, because
NAis a trap for the next anchor added without rounding.Was this load-bearing?
No, and the margins say so:
The value is preventative. If anyone tightens
mem.normalize_headfrom 1e-13toward exactness — the natural direction for a fidelity project — it would begin
failing for a serialization reason rather than a numerical one, and that is a
hard failure to diagnose. Now it cannot.
One genuine reader-facing fix
The extra precision exposed a display bug in the out-of-core report: a hard
22-character slice rendered
1.0153110547861388e-06as1.0153110547861388e-0, dropping the exponent's last digit. A reader would takethat for 1e-0 — a six-order-of-magnitude misread of a number the vignette
presents as evidence. Values are now shown whole or elided with an ellipsis, and
a 1-element list from the R side is unwrapped so the number gets the column
instead of the brackets.
Verification
All four R scripts re-run; all four comparisons re-run afterwards.
1168 passed, 25 skipped.
tutorials/figures_spatial/r_reference.jsonis the onetracked R reference, so its regeneration is included in the diff.
🤖 Generated with Claude Code