Conversation
A CBOR sequence has no way to reach item n except by skipping the n-1 before it. Fine while tailing a log, useless while seeking: reaching the last of 200 000 items measured 9 896 us by skipping and 0.43 us through an index. `write-seq!` with `:index N` seals a sequence with ONE extra item: <item> ... <item> d9 9ae3 84 <stride> <total> <int32 offsets> 48 <8 bytes> At the END, because offsets are only known once the items are written -- a leading index would mean buffering the whole sequence in memory before emitting a byte, which is fatal for a log. ZIP's central directory and Parquet's footer are at the end for the same reason. A foreign reader also sees every real item first and the metadata last. Found through its own last element rather than a magic trailer: an 8-byte byte string always encodes as 0x48 plus 8, so a sealed file ends with 9 predictable bytes however large the offsets array is. Those bytes are ordinary CBOR. The file stays a valid sequence that cbor2 and ciborium read in full, getting the data items plus one tagged value they can ignore -- verified in the tests. The pointer verifies as well as locates: it is both where the index starts and how long the data section is, so a reader that seeks there and does not find tag 39651 knows the index is stale and scans instead. THE INDEX IS NEVER LOAD-BEARING FOR CORRECTNESS. That is what the tests pin: a corrupted pointer, a truncated file, an appended-to file and a file that merely ends in the right 9-byte shape all still produce right answers, the last one being the false-positive case the three checks exist for. Truncation surfaces as a TYPED error, not a fabricated answer. Detecting the index is not only about speed -- it is itself a top-level item, so without recognising it `nav/items` would yield it as though it were data. Stride is a parameter because the sweet spot moves with item size: each entry costs 4 bytes while a lookup scans up to N-1 items. On 200k ~40-byte items, stride 1 costs 10.1% of the file and stride 16 costs 0.6% for 2.4x the lookup. No default is worth baking in. 39651, not 39650: SHAPES.md already specifies 39650 for scattered shapes. Both documented there, and IANA-REGISTRATION.md now records that the provisional-tag debt is two numbers rather than one. Scope, stated in SHAPES.md rather than left to be discovered: this indexes top-level items only. Descending into a map is still a scan, and that scan is linear in the map's width -- 0.35 us at 10 keys, 276 us at 5 000. A hierarchical index would fit this mechanism unchanged; the open question is what to index, which is a schema decision, not an encoding one.
Tag 39651 now carries a node per CONTAINER as well as the sequence offsets.
A node is the byte offsets of a container's entries, so an array indexes
positionally in O(1) and a map with sorted keys binary-searches in O(log n),
comparing ENCODED key bytes -- no key decoded, no value touched.
keys overhead scan indexed
100 3.4% 4.23 us 2.09 us
1000 1.4% 26.1 us 0.79 us
10000 1.2% 175.1 us 0.74 us 237x
TWO ALGORITHMIC FIXES, both found by measuring rather than reasoning.
The walk now returns each value's END OFFSET instead of calling skipFrom per
entry. skipFrom is O(subtree), so an entry-at-a-time walk re-walks everything
beneath each level: O(n^2) in depth, measured at 486 ns/byte against a skip's
1.3-2 and 27x slower on a 200-deep document. One descent visits each byte once
and the node falls out of it. Building an index costs 1.1-1.4x a plain encode,
which is what makes deriving it from encoded bytes -- rather than hooking the
writer's hot path -- the right trade. It also means any already-encoded value
can be indexed after the fact.
node-for allocated a map per level per lookup, which made DEEP paths slower
with an index than without. It returns an int now.
WHERE IT STILL LOSES, and this is in the benchmark output and SHAPES.md rather
than buried: finding a node is itself a binary search over the container list,
once per level. For containers that are narrow AND whose entries are cheap to
skip, walking beats jumping -- 64-key levels measured 2x SLOWER indexed.
:index-min above such widths is the fix.
:index-min default was 2, which was badly wrong: real data is mostly small
containers and each costs an offset, a count, a slot and a flag. On 2000
two-field records, indexing everything cost 76% of the file and indexing only
8+ cost 1.3% -- and was faster, the smaller index fitting in cache. Now 16.
Tags are DESCENDED THROUGH, not stepped over. A set is tag 258 around an array,
a record tag 27 around [name, map]; skipping tags left exactly those uncovered,
and descending is free since skipFrom walks the same bytes anyway.
A bounded-scan bug the tests caught: a key missing past the last anchor walked
a full stride off the end of the container. The last anchor covers a remainder,
not a stride.
Same guarantee as everything else here: the index is an optimisation, never
load-bearing. Missing, stale or corrupt, every path returns the same answer.
`bin/ci lint` fails on WARNINGS, not just errors. I had been checking with `clojure -M:lint` and grepping for "error", which reported clean while CI's jvm job went red on a redundant-let warning. Run the CI stage, not the alias.
That paragraph was written when the index was sequence-only. The container nodes shipped in the same PR, and I appended a section for them without removing the old one -- so the document asserted both that it indexes only top-level items and, forty lines later, that it carries a node per container. Replaced with a pointer to the section that supersedes it, and noted that the sequence offsets are the node at sentinel offset -1, so the two are one mechanism rather than two. The PR description carried the same stale claim and has been updated.
Tag 27 is CBOR's registered extension point for named types, and boring already reserves slash-bearing names under it (clojure/sorted-map, java/period). The index is now one more. Measured, because the two tags this branch touches have OPPOSITE answers and the difference is occurrence count: payload shaped arrays cost as a tag-27 name datom-maps-200 1 +0.4% 100 small tables 100 +19.6% nested tables x500 500 +35.2% the index item 1 PER FILE +0.05% So the index moves and 39649 stays. A name costs 14 bytes, which against one occurrence per file is nothing, and it buys: one registration obligation instead of two, self-description (cbor2 sees the string, not an unregistered number), and strictly narrower false-positive detection -- a stray file must now end in the right 9 bytes AND point at tag 27 AND carry this exact name. 39649 keeps a number for the mirror-image reason: shaped arrays occur per array, and :shapes exists to shrink documents. Paying up to a third of the file back would defeat it on precisely the data it is for. That rationale was missing from the docs -- the question "why not tag 27?" had no written answer -- and is now in SHAPES.md and IANA-REGISTRATION.md, which is the document that will be read when the registration is actually filed. Detection reads the name before decoding, so a false positive cannot even get as far as parsing garbage. `frame-payload`/`frame-name` rather than `record-fields`/`record-type`: an unregistered tag-27 frame decodes to an UnknownRecord when its payload is a map and a TaggedLiteral otherwise, and this payload is a vector. Both accessors already existed for exactly this. INTEROP.md lists `boring/index` among the reserved names, noting it is the one a foreign reader can ignore with NO loss -- it is an optimisation, never data.
Offsets inside a container ascend, so consecutive differences are small and
nearly uniform while the absolutes are large and unbounded. Each slot now goes
out as differences from the previous entry -- from the container's own offset
for the first, or 0 for the sequence node, whose sentinel -1 is not a position
-- in the narrowest of a byte string, sint16 (tag 77) or sint32 (tag 78).
NO NEW FORMAT SURFACE. All three types already round-trip through the codec,
and the CBOR element type IS the width declaration, so there is no per-entry
flag of the kind PostgreSQL needs for its JEntry array. Postgres reads offsets
in place out of a TOAST'd datum and pays a prefix sum per probe; we materialise
the index once when it loads and expand there, so lookup_map, nth and node-slot
still read a plain int[] and nothing on the hot path changed. The binary-search
cost I set out to measure does not exist -- it was a consequence of their
constraint, not of delta encoding.
A reader must therefore treat an int32 slot as deltas too: absolutes and deltas
are indistinguishable on the wire. That is exactly why this had to land before
the format was published, and merging to main publishes it -- `version` is
0.1.<git-count-revs> and tools/deploy runs on every push to main.
WHAT IT IS WORTH, on 200 000 items of ~36.8 bytes:
stride slot type index overhead us/seek
-- (no index) -- -- 10 600
1 byte string 195 KB 2.72% 0.2
8 sint16 49 KB 0.68% 0.6
16 sint16 24 KB 0.34% 1-2.5
64 sint16 6 KB 0.09% ~4.2
Absolute int32 offsets cost 781 KB at stride 1, 10.9% of that file; deltas cost
195 KB. That 4x is the point: it makes stride 1 -- an index with no scan
component at all -- a defensible choice rather than a curiosity, which is the
precondition for indexing by default. At stride 16, where the index was already
0.38% of the file, this changes nothing anyone would notice; the earlier claim
that it was worth 25x measured the index against itself on synthetic data and
answered a question nobody had.
Narrowing also breaks proportionality: doubling the stride halves the anchor
count but can double the width, so size falls in steps and there are bands
where a denser index is free.
The stride table was never reproducible -- it was measured ad hoc and written
straight into SHAPES.md. It is now a benchmark, `-m nav index`, which also
prints the unindexed baseline the whole feature exists to remove. Seek is
separated from decode because only the seek is the index's doing: materialising
the item is a ~0.3-0.5 us floor at any stride, which dominates at stride 1 and
vanishes by stride 64. The large-stride rows are quoted loosely on purpose --
stride 16 moved between 1.1 and 2.4 us across runs and 64/256 swapped order,
because the index competes for cache with the data.
Also adds `mmap/mmap-items`, which the new test needed and which was missing:
`mmap-source` returns a cursor, the single-value shape, and there was no
equivalent for a sequence -- the headline case for an index over a mapping. The
mmap test is what pins delta expansion through the segment accessor rather than
the byte[] fast path, where a disagreement would not throw but would silently
seek to the wrong offset. It asserts 500 items rather than 501, so it fails if
the index is merely not detected.
`read-index` required a positive back-pointer. An empty data section puts the index at byte 0, so sealing zero items produced a file whose own index was refused -- and the trailing frame was then walked as though it were data. `write-seq!` of nothing read back as one `boring/index` item instead of none. Found by testing degenerate sizes rather than by review: 0, 1 and 2 items at two strides, which is now a test. One and two were always fine; only zero has a zero-length data section. `>= 0` does not widen the false-positive surface in any way that matters. A stray file would have to end in the right 9 bytes, have all eight of them zero, AND begin with a tag-27 frame carrying the name -- the name check does the work here, as it does for every other pointer value. Also tests a container holding one 50 KB value among 39 small ones. Widths are chosen per slot from the largest delta, so an outsized entry pushes its whole slot wider; getting that bound wrong would corrupt only the entries after the big one, which a uniform fixture cannot catch.
whilo
changed the title
The pointer and name checks establish that something INTENDS to be an index, not that its payload is well formed. A frame that passes both and then holds the wrong shape threw at the caller of `nav/source`, which contradicts the invariant the rest of this branch is built on: the index is an optimisation, and the honest response to one we cannot use is to walk. This matters more now that slots are expanded eagerly when the index loads. Before, a bad slot only broke lookups that reached that container; now it would take down a cursor that never went near it. The test carries a CONTROL, because the obvious version of it passes for the wrong reason: if the hand-built pointer were wrong, detection would fail early, the reader would scan, and the assertion would hold without the guard ever running. So the same pointer arithmetic is repeated with a USABLE payload and asserted to be detected -- 3 items, not 4 -- which pins that the bogus case fails in expansion rather than never getting that far. Also documents the limit of the never-load-bearing claim instead of leaving it implied: there is no checksum, so a payload corrupted into something that still parses will be believed and will seek to the wrong place. Detection catches malformed indexes, not subtly wrong ones. That is the same exposure the data section already has -- CBOR carries no checksum either -- so the index is no more fragile than the bytes it describes, but the asymmetry was worth writing down rather than letting a reader infer a guarantee that is not there.
`tools/format` went red: wrapping the payload in try/catch left its body at the old indentation, and cljfmt's fix reindented the forms but not the comments sitting between them, so the result was correct and unreadable -- ten levels of nesting with every comment two columns off from what it described. Extracted `index-payload`, which is the natural seam anyway. `read-index` now does DETECTION -- the tail shape, the pointer, the name, all of which answer "does something here intend to be an index" -- and `index-payload` answers the different question of whether that payload can actually be used. The try/catch belongs to the second, which is why it reads better as its own function than as another level of nesting inside the first. No behaviour change: same 155 tests, and `clj -M:format` is clean.
Indexing a 50 000-record log cost 3.1x a plain write, and I had inferred the
cause was the second pass over the bytes -- so a sequence-only mode, which
needs no pass at all, would be the way to make it affordable. The profile says
otherwise: `skipFrom` and `skipValue` together are 1.6% of samples. The walk
was never the problem, and that inference was wrong.
What it actually was, both allocation:
1. `scan-index` shaped a full result per item -- an ArrayList, a sort, four
sequence traversals and a four-entry map -- and `write-seq!` called it once
per item, then copied out of it into a sequence-wide accumulator. For a
typical log record, whose containers all sit below `:index-min`, every bit
of that described ZERO nodes. Split into `scan-into!`, which appends
straight onto the caller's accumulator, and `nodes->index`, which shapes
once per sequence.
2. `index-walk` allocated `(int-array n)` for EVERY container before testing
`min-entries`, then copied every stride-th element into a second array. So a
document of small maps allocated one throwaway array per map. It now
allocates only for containers it will keep, and only one slot per ANCHOR --
which also removes the copy. This is why raising :index-min barely helped
before: the arrays were allocated either way, just not kept.
Measured with a harness that warms every path before timing any of them and
interleaves the cases, because the first one measured was otherwise penalised
hard enough to invert the ranking -- an earlier run had "no index" slower than
stride 1, which is what prompted rewriting the harness rather than the code:
before after
no index 19.0 ms 19.8 ms
stride 1 66.3 ms 45.8 ms
stride 16 59.1 ms 36.9 ms
Overhead over an unindexed write drops from +211% to +86%.
WHAT IS LEFT, and it is one thing: boxing. 67% of remaining allocation is
`java.lang.Long` and 9% of CPU is `Long.valueOf`, from `index-walk` recursing
through six untyped parameters -- four of them longs, which is one more than
Clojure can pass primitively alongside the Reader and the accumulator. The walk
belongs in Java next to the Reader it drives; that is where the rest of the
+86% is.
The index no longer comes from walking the encoded bytes. The writer already
knows `pos` -- the offset it is about to write to -- and knows a container's
entry count before emitting a byte of it, so the nodes fall out of encoding.
That the writer CAN do this is the mirror image of why the reader could not.
Nothing in the index needs a subtree's LENGTH, only where its entries start, so
there is nothing to back-patch. A length-prefixed format would need a second
pass or back-patching here; CBOR's element counts make the write side free and
the read side expensive, and this takes the good half of that trade.
WHAT IT COSTS, 50 000 records through a BufferedOutputStream to a real file:
byte walk captured
no index 20.1 ms 20.1 ms
stride 16 36.3 ms 19.7 ms +80% -> +3%
stride 8 37.2 ms 20.6 ms +85% -> +5%
stride 1 42.9 ms 25.2 ms +113% -> +30%
Stride 1's remainder is not capture -- it is `delta-slot` and the encode of a
50 000-entry slot at the end, which is per-anchor work the walk paid too.
TWO WRONG TURNS, both caught by measuring rather than by review.
The first instrumented `writeMapValue`, which is correct and was never called:
the general dispatch carried an INLINED COPY of its loop, so every plain
Clojure map came back unindexed while the named method looked right. The Map,
Set and Collection cases now delegate instead of duplicating, which is why the
diff touches dispatch at all.
The second replaced the ArrayList of boxed sequence offsets with a Clojure
volatile holding an int[], to kill one Integer.valueOf per item. It made stride
1 SLOWER -- 40% to 53% -- because three volatile reads and two volatile writes
per anchor cost more than the allocation, a volatile write being a store
barrier. Moving the same state into a plain Java field on the writer, which is
not thread-safe anyway, took it to 30%.
THE GATE IS boring/writer-index-test, generative, against `build-index` as the
reference implementation. It found the one real divergence within a minute:
the walk indexes containers on the WIRE, and boring puts several there that no
user wrote -- a sorted-map is tag 27 around a two-element [name, map], a shaped
array is [keys, rows]. The walk cannot tell those from a user's own two-element
vector; the writer knows and skips them.
That is a subset, and a subset is legitimate -- the index is never load-bearing,
and a node for [name, map] is pure overhead. So the pinned contract is: every
captured node is byte-identical to one the walk found (always), and the two
agree completely once :index-min excludes frames. Every frame boring emits has
3 entries or fewer, so 4 is the boundary and the default of 16 clears it.
Worth noting how it shrank: to `(sorted-map)`, which prints as `{}` and is
indistinguishable from a plain empty map. Every attempt to reproduce it by hand
from the shrunk output passed.
`build-index` stays. It is the only way to index bytes somebody else wrote, and
it is now also the oracle this is tested against.
SHAPES.md still said the index is "derived by walking encoded bytes, not by hooking the writer -- so the writer's hot path is untouched", which was the design rationale right up until the writer started capturing it. It also quoted 1.1-1.4x a plain encode, measured on `encode-indexed` for a single value rather than on a sequence write, where the real figure was +80%. Now describes both: `write-seq!` captures while encoding (+3% at stride 16), `build-index` walks (+80%) and stays, because indexing an already-encoded value has no other path and it is the oracle the captured index is tested against. Also writes down the deliberate divergence, which was previously not documented anywhere a reader would find it: the walk indexes containers on the wire, including frames boring emits that no user wrote, and the writer skips those. Subset, not disagreement, with the boundary stated -- every frame boring emits has 3 entries or fewer, so :index-min 4 separates them and the default of 16 clears it. And the walk's floor, which is the thing worth knowing if anyone tries to optimise it again: ~31% of encode time, because stepping over a subtree IS walking it. Capturing avoids that floor rather than improving on it.
Two independent reviews of this branch found that the claim the whole feature rests on -- the index is an optimisation, never load-bearing -- was false in six ways. Four of them fire on ordinary data at the shipped defaults, with no corruption involved. Each is now a regression test in boring/index-robustness-test, and 7 of its 8 original cases fail without these fixes (the 8th is a control that must pass either way). 1. SORTEDNESS WAS DECIDED FROM A SAMPLE. `sorted` licenses `lookup-map` to binary-search the anchors and then scan only the stride it lands in, which is valid just when the whole container is ordered. It was set by comparing the ANCHORS -- and ascending anchors do not imply an ordered container. At the default stride of 16 a map of 17-32 entries has two anchors, so an unordered map was marked sorted about half the time and a key that EXISTS returned nil: 105 of 200 random 20-entry maps had at least one, 797 wrong lookups in 4000. Now every adjacent key pair is compared, in both builders. The differential test could not see this: it compares the two builders against each other and both had it. An oracle that shares the bug proves nothing. The new test's oracle is the decoded map. 2. `nth` THREW NPE whenever the index had no sequence node -- which is every index `encode-indexed` or `build-index` + `seal-index!` produces, since only `write-seq!` emits the sentinel -1. The 3-arity not-found form threw too, so there was no safe way to call it, while `seq` and `reduce` worked. 3. `:canonical-rfc7049` SORTS LENGTH-FIRST, readers compare bytewise. I had just changed `writeMapCanonical` to assert `sorted = true` "by construction", without asking which comparator had run. 11 of 20 keys silently returned nil on a map mixing short text keys with large integer keys. 4. APPENDING A SEALED BATCH LOST THE EARLIER ONE, silently. `write-seq!` counts from 0, so its back-pointer is chunk-relative; concatenate two sealed batches of equal data length and the trailing pointer lands on the FIRST batch's index frame -- a real tag-27 `boring/index`, so every check passed and the reader stopped at the first chunk's end, reporting 100 of 200 items with no error. A live index must now END AT THE FILE'S END, which a stale one does not, so it is refused and the reader scans. 5. A CORRUPT BACK-POINTER COULD THROW. The head parser rejects reserved additional-info 28-30, and probing was outside the try that was supposed to guarantee the fallback -- so `nav/source` threw on a file whose only fault was a damaged pointer. Roughly 3 in 256 of random corruptions, and binary payloads carry 0xDC routinely. The old test flipped a byte to 0x7F, which lands out of range and never reaches the parser. 6. THE PAYLOAD WAS NEVER CHECKED FOR CONSISTENCY. Detection proves something MEANT to be an index; nothing proved its parts agreed. A frame that merely decoded was trusted, giving raw IndexOutOfBoundsException at the caller of `get` or a wrong subtree. Now checked: the four arrays are the same length, containers ascend (`node-slot` binary-searches them), counts are non-negative, each slot's length matches its count and stride, and every anchor ascends, lands inside the data section and starts after its own container's header. This caught a malformed fixture in the existing suite immediately: a control payload declaring three anchors for three items at stride 16, which needs one. It only ever passed because nothing checked. ALSO, from the same reviews: - An index offset past 2 GiB silently became NEGATIVE, so containers stopped ascending and `nth` seeked backwards. Refused now: an unindexed file is correct, a wrongly-indexed one is not. - A failed indexed `write-seq!` left capture running with a stale base, so every later encode on that (deliberately reused) writer retained a node per container, invisibly and forever. Now turned off on the way out. - `idxReset` forgot the count but left every anchor array reachable. It nulls them, as the stringref table beside it already did. - `setIndex` now starts a fresh capture, with `idxBase` for moving between items of one sequence. Index state must survive `reset()` -- `write-seq!` resets per item while nodes accumulate across the sequence -- so the fresh start needed somewhere else to live. Write cost is unchanged at the defaults: +4% at stride 16, +5% at stride 8. The per-entry key comparison shows up only at `:index-min 2`, where every small map is indexed, at 177% -> 191%. Correctness at that setting is worth more.
The same two reviews turned up defects outside the index. Two are on `main` and
so are in a released version; none is caused by the index, but all of them ship
with it. Each is a regression test in boring/index-robustness-test, and all five
new cases fail without these changes.
ON MAIN, IN A RELEASED VERSION:
1. `boring.nav` DROPPED EVERY DECODE OPTION. `nav-of` built its Reader and never
configured it, so `opts` reached only the encode side used for key probes.
`:registry` was ignored -- a registered record came back as a raw tag-27
frame instead of the type -- directly contradicting `nav/value`'s docstring
("through the ordinary reader -- same registry, same records, same
everything") and `source`'s ("`opts` are the decode options realisation will
use"). A caller's `:max-depth`, which is a SECURITY bound, was not enforced
on the navigation path at all. `configure-reader!` is now ^:no-doc public so
nav can apply exactly what `decode` applies.
2. NESTED TAGS WERE NOT BOUNDED BY `:max-depth`. Arrays and maps charge the
depth budget; the tag branch of `skipStructural` recursed without it, so tag
nesting was the one shape maxDepth did not bound -- `c0 c0 ... 00` skipped
happily at maxDepth 1 and blew the stack with enough of them. `read` charges
for tags, so skipping has to agree, or the cheap path routes around the
expensive path's limit -- and skipping is what navigation does.
ELSEWHERE:
3. A FAILED READ POISONED THE READER'S DEPTH. `enter()` incremented before
throwing, and only array/map unwind in a finally, so a rejected read
permanently consumed a level of budget. `boring.nav` shares one Reader across
every lookup, so a later, perfectly shallow read failed because of an earlier
one. Checked before incrementing now, and `readFrom`/`skipFrom` restore depth
alongside pos -- these are the navigator's whole interface to the Reader and
it calls them on values that may legitimately fail.
4. CANONICAL MAPS COULD EMIT DUPLICATE KEYS. Distinct keys can encode
identically -- Long 1 and BigInteger.ONE are both `01` -- and the result is a
document boring's own decoder rejects as :boring/duplicate-map-key. Canonical
SETS have always checked this; maps did not, so the same hazard produced
unreadable output instead of an error.
5. `canonicalSubWriter` DID NOT INHERIT `encodeFallback`, while its own comment
claimed "EVERY behaviour-affecting option is inherited". An :encode-fallback
configured to rescue an unsupported value worked everywhere except in a map
key or set element under :canonical -- where those are pre-encoded -- so the
option silently did not apply exactly where the document is hardest to
repair by hand.
6. A COLLECTION WHOSE size() DISAGREED WITH ITS ITERATION was written anyway.
The head is emitted from size() before the entries, so a mismatch is a
malformed document either way; with an index it is worse, because the anchor
array is sized from the same number -- an over-run walks off it and an
under-fill lets the NEXT item be swallowed as the missing element, which in
an indexed sequence was the index frame itself. Refused now, with the
RandomAccess path exempt since it indexes by position.
7. `write-seq!`'s 3-ARITY IGNORED THE WRITER'S OPTIONS, unlike `write-to!` and
`encode-into!`. A writer built `(writer n {:stringref false})` -- exactly what
a navigable file needs -- silently emitted stringref output through that one
entry point, and `boring.nav` then refused to read what it had just written.
Not fixed, and deliberately: the index-array growth arithmetic (`idxN * 2`)
overflows at 2^30 entries, and a rectangular matrix's element count can wrap
before widening. Both need multi-gigabyte live inputs and will OOM first; the
2 GiB offset guard added earlier fires long before either.
Charging tags to the depth budget in `skipStructural` closed a hole -- `read` already charged for them, so skipping was the more permissive path and routed around a documented SECURITY bound. The risk in fixing it is the mirror image: skip must not now reject anything `read` accepts, because navigation skips constantly and a regression there would surface as valid documents becoming unnavigable. Checked across seven nesting depths and four max-depth settings on values built from four alternating shapes, two of which are tagged (a set is tag 258, a sorted-map tag 27), so both tagged and untagged levels are exercised. The two paths agree everywhere.
`((0 - 1) / stride) + 1` is 1 in Java, because integer division truncates toward zero, and the Clojure walk's `(max n 1)` produced the same 1. So an empty container claimed one anchor, the loop never wrote it, and the slot kept a phantom offset of 0 -- pointing at the start of the document, and marked sorted. Only reachable with `:index-min 0`, which nobody should set, but it is a wrong answer waiting rather than a missed optimisation. It also mattered indirectly: the new payload validation checks that a slot's length matches ceil(count / stride), so a single phantom anchor made the reader refuse the ENTIRE index and fall back to scanning. Fixing the cause is better than having the validation paper over it. Both builders, since both had it.
A review of the previous round's fixes found that three of them introduced or
left new bugs, and that several of the regression tests were falsely reassuring.
That is the round working as intended, and it is the second time today a fix of
mine caused a fresh defect.
WHAT I BROKE:
1. TAG DEPTH ACCOUNTING. Charging tags to the depth budget so `skip` matched
`read` made skip STRICTER instead: `readTagged(258)` parses the set's array
head inline without entering, so read charges for the tag but not the
container beneath it, while my mirrored skip charged for both. An ordinary
`#{}` was refused at a depth decode accepted -- navigation rejecting a
document that decodes fine.
Mirroring read's accounting case by case is not maintainable; there are as
many rules as tag readers, and I found three contradictory ones by trial. The
tag chain is now consumed ITERATIVELY and bounded by its LENGTH, which fixes
the original stack overflow outright rather than capping it, and can only
make skip laxer than read -- never stricter. Safe by construction rather than
by my enumerating read's quirks correctly.
The property test asserted exact parity, which was the wrong specification
and is what sent me chasing those quirks. What matters is one-directional:
skip must not reject what read accepts.
2. THE EMPTY-ANCHOR FIX left indexed empty-map lookup crashing. `lookup-map`'s
binary search assumes at least one anchor, and `(max 0 (min (dec 0) hi))` is
still 0, so `aget` threw at the caller of `get`. My test realised the empty
map but never looked INSIDE it.
3. INHERITING encodeFallback opened a stack overflow: the scratch writer did not
inherit the re-entry guard, so a fallback returning a value that still
contains the unsupported object recursed forever instead of raising the typed
error. `depthOffset` was not accumulated either, so nesting through key
position got a fresh budget each hop.
WHAT WAS STILL INCOMPLETE:
4. A CRAFTED INDEX COULD STILL MISDIRECT. Anchors can ascend, sit in range, and
not be entry boundaries: three 4-byte items at 0, 4, 8 with anchors 4, 8, 9
passed every structural check and made `nth` return the NEIGHBOURING value.
Closed by two O(1) reads off the container's own head -- the node must
describe a container that is really there with the count it claims, and its
first anchor must be that container's first entry.
5. SIZE-MISMATCH CHECKING covered only the collection paths. The ordinary
non-indexed map, records, and the canonical staging arrays had none, and
canonical staging threw a raw ArrayIndexOutOfBoundsException from an array
sized by size().
AND THE TESTS THAT WERE LYING: the empty-container test never looked inside the
empty map and wrapped its assertions in a `when` that made them vanish; the
sortedness tests exercised only the Clojure walk, never the Java writer capture;
the depth generator never produced an empty container, which is exactly where
the paths diverged; there was no canonical :encode-fallback test; the size test
covered AbstractCollection only. Each gap is now closed, and the depth failure
shrank to `(sorted-map)` -- which prints as `{}`, the same disguise that cost me
an hour yesterday.
The general property the tag-chain rewrite could have broken. That loop reads a head to see whether the chain continues and, when it does not, pushes the byte back with `pos--` -- correct only because `u8()` is `b(pos++)` and advances exactly one byte. Had it been wrong, skipping would land MID-ITEM, and every later offset in a sequence would be garbage. Nothing existing would have caught that. Decoding never uses skip, so every round-trip test passes regardless; the index tests use offsets the writer produced rather than offsets skip derived. This asserts the one thing that ties them together. Generated over four profiles rather than enumerated, because the shapes that matter are the tag-wrapped ones -- sets (258), sorted maps and sets (27), ratios (30), uuids (37), shaped arrays (39649). 4000 values across four profiles landed exactly, zero mismatches.
…ing tests
A third review found that the previous commit's `write-seq!` 3-arity fix THREW
for three of the five profiles. `writer-opts` hands back an ALREADY-RESOLVED map
with `:profile` stripped; delegating to the 4-arity re-resolved it, so
`resolve-opts` saw `:canonical` without the profile that licenses it and raised
`:boring/incompatible-options` -- on a public entry point, for `:archival`
(shipped one commit earlier), `:canonical` and `:canonical-rfc7049`.
My test for that fix used `{:stringref false}`: the one profile where the
round-trip through `writer-opts` is a no-op. It now covers all five.
Resolution happens ONCE now, in whichever arity was called, which is what
`encode-into!` already did -- the model the previous commit cited without
following.
AND TWO TESTS THAT PASSED WITH THEIR FIX REMOVED:
- `a-failed-read-does-not-poison-the-readers-depth` built a FRESH Reader on
every iteration, so it never exercised the shared-Reader leak it is named for.
It now reuses one Reader across the failures and the later shallow read.
- `empty-containers-get-no-phantom-anchor` drove only the Clojure walk.
`Writer.anchorCount` is the other half of the same fix and is what
`write-seq!` uses -- the primary index producer -- and deleting its guard left
the suite green. Covered now, and verified to fail without it.
The method that found these is worth keeping: revert ONE fix, run the suite, and
require its test to fail. I had been reverting all of `src/` at once, which only
proves the batch is covered, not each fix. Two of five were not.
A caution learned the hard way in the same sweep: `git checkout --` on a file
carrying an uncommitted fix silently discards it. Commit before reverting
anything.
…dary The third review's remaining two findings are not code defects but an overstated claim. The branch asserts, in SHAPES.md and in the robustness suite's own docstring, that the index is never load-bearing for correctness -- flatly, with no qualification. That is true for a missing, stale, truncated or randomly corrupt index: each is detected and costs only a scan, which the tests pin. It is FALSE for a crafted one. The reader validates that a node's parts agree, but it cannot check that EVERY anchor is a real entry boundary without an O(n) walk per container, nor that `sorted` is truthful without reading every key -- both exactly the work the index exists to avoid. Demonstrated: anchors 1, 5, 8, 9 over entries at 1, 3, 5, 8 pass every check and make `nth` return neighbouring values; flipping one `sorted` bit on an otherwise genuine index loses 19 of 20 keys. Closing that by validation would defeat the feature, so the honest move is to say what the guarantee covers. The index frame is a trust boundary: integrity of the index is integrity of the document. That is the same exposure the data section already has -- CBOR carries no checksum, and anyone who can rewrite the index can rewrite the data -- but the sentence claimed more than it delivered and now does not. Also corrects `scan-into!`'s docstring, which still described `write-seq!` as calling it once per item. It has not since the writer began capturing the index; its only caller is `build-index`, always with base 0, which is why the unchecked narrowing there is safe.
The previous commit qualified it in SHAPES.md and the robustness suite, but the same unqualified sentence was still in doc/STORAGE.md, boring.nav's own docstring and writer_index_test.clj. A guarantee that holds in one document and overclaims in three is not qualified; a reader takes whichever they open first. All four now say the same thing: detection covers a missing, stale or randomly corrupt index, and the index frame is a trust boundary against a crafted one, with SHAPES.md carrying the reasoning.
…fety only
A fourth review found three more defects, and the worst is the third instance of
one root cause.
1. SHAPED ARRAYS decoded but would not navigate. Tag 39649's reader consumes the
outer, keys, rows and row heads INLINE and charges depth only for the tag,
while a generic skip must recurse into each -- so `[{"a" 1} {"a" 2}]` encoded
with `:shapes true` decoded at `:max-depth 1` and threw from `byte-span`.
That generalises, which is the point. Any tag reader parsing its payload
inline charges nothing for the containers inside it. The gap is three levels
per shaped array and shaped arrays nest, so it is unbounded and no constant
slack closes it. Three attempts to mirror `read` each broke a different
value -- charging for tags refused `#{}`, not charging for empty containers
accepted `(sorted-map)` where decode refused, charging per container refuses
shaped arrays -- and the fourth stops trying.
Skip's bound is now a STACK bound: max(maxDepth, 1024), on its own counter.
Deliberately LAXER than read, because stricter is the direction that breaks
navigation, and a document read cannot handle without overflowing its own
stack is not one skip needs to accept. Semantic depth is still enforced on
every value the caller realises; it was never skip's job.
2. A VALID INDEX MADE READING FAIL. The index frame is a fixed nested shape, so
decoding it costs four or five levels however shallow the data is. Read
against the caller's `:max-depth`, an indexed `[1]` failed to decode its own
index, forgot it, then walked into the frame as data and raised -- worse than
having no index at all. The frame now decodes against its own budget; the
caller's limit is a bound on THEIR data and is still applied to everything
they realise.
3. `build-index` STILL OVERFLOWED THE STACK on a tag chain. `index-walk`
recursed once per tag, and its comment claimed the recursion was bounded by
the decoder's maxDepth -- false, because positional reads never touch the
Reader's depth. 20 000 bytes of `c0` through the public `build-index` was a
StackOverflowError. Consumed iteratively now, as `skipStructural` already
was; collapsing a chain is equivalent because a tag's extent IS its
payload's.
Also covers three sub-fixes the review found had no test that could distinguish
them: accumulated `depthOffset` through canonical map KEYS, canonical set size
checking, and the shaped-array case above. And `nested-tags-are-bounded` now
asserts the new bound rather than the old one -- a 40-tag chain at `:max-depth 4`
is accepted ON PURPOSE, which is precisely the case that used to be refused.
1. THE INDEX FRAME WENT OUT UNDER THE WRITER'S OPTIONS, NOT THE CALL'S.
`seal-index!` emitted it through `write-to!`'s 2-arity, which resolves
`(writer-opts w)`. So `write-seq!`'s 4-arity wrote the DATA with the caller's
opts and the FOOTER with the writer's -- and the documented shape,
`(write-seq! (writer 4096) events out {:stringref false :index 8})`, put the
frame inside a stringref namespace, which `nav` refuses to recognise. The
index was silently dead (27x slower lookups) and the frame came back as a
PHANTOM TRAILING ITEM in every log.
Release-breaking, and pre-existing since the index first landed. Invisible to
every test and every doc example because all of them happen to build the
writer with the same options they pass. `seal-index!` is public, so it gains
an opts arity rather than only being fixed at its callers.
The first attempt at this reintroduced the double-resolution bug from the
previous round -- `write-to!`'s 3-arity resolves again, which throws under
every profile that locks a key. It writes through `write-root!` now, which
takes options already resolved.
2. A DAMAGED INDEX COULD STILL THROW AN UNTYPED EXCEPTION AT `get`. Validation
proves the FIRST anchor of a node is a real entry; proving it of every anchor
would cost the walk the index exists to avoid. A middle anchor pointing
mid-item therefore survives, and the walk from it read a garbage head and ran
off the buffer.
Three separate paths needed guarding, and I found the third only by looking
at the stack rather than reasoning about it: the scan's starting point, the
binary search's probe, and -- the actual one -- the walk itself, where a
start can be in range while the item's DECLARED length runs past the end, so
the throw happens inside `skipFrom` before any check on its result could see
it. Out-of-range now reports a miss.
Pinned by mutating EVERY byte of a real indexed document to four values and
requiring that no lookup throws anything untyped: 4 139 assertions.
3. THE DEPTH TEST WAS VACUOUS FOR THE SECOND TIME. Its first version built a
fresh Reader per attempt; its repair called `.reset`, which sets `depth = 0`
and wipes the leak the assertion exists to observe. Both values now live in
ONE buffer and the shallow one is read at its own offset with no reset --
which is how `boring.nav` actually uses a Reader, sharing one across every
lookup. Verified: reverting the fix now fails it.
Worth recording that `enter()`'s check-before-increment is NOT independently
observable, because `readFrom`/`skipFrom` restore depth and `reset` zeroes
it. It is defence in depth and the restore is what carries the behaviour; a
test that appeared to pin it alone would be theatre.
4. "RANDOMLY CORRUPT" WAS THE WRONG WORD, in five places -- and I had propagated
it to four of them a commit earlier while claiming to be qualifying the
guarantee. Exhaustive single-byte mutation of a real index frame returns a
silently WRONG ANSWER 2.1% of the time, and a ONE-BIT flip of the `sorted`
byte loses 15 of 20 keys with no error. The line is not crafted-vs-random, it
is structurally consistent or not, and ordinary bit rot lands on the wrong
side. Corrected in SHAPES.md, STORAGE.md, boring.nav, writer_index_test and
bench/nav.clj.
Also: the size-mismatch test's "indexed" row was not indexed -- `encode` never
calls `setIndex`, so it duplicated the plain row. It goes through `write-seq!`
with `:index` now, which is the path that actually captures.
Two defects found by stress rather than by reading, and neither is about the index -- both are contracts `boring.nav` was not keeping. 1. POSITIONAL READS THREW UNTYPED ON DAMAGED INPUT. `decode` reports `:boring/truncated-input`; `boring.nav` threw a raw ArrayIndexOutOfBoundsException for the SAME corrupted byte. Measured on an UNINDEXED document -- 415 untyped throws in 5360 mutations -- so this is the navigator's behaviour with damaged data generally, not anything the index introduced. `read` gets typed truncation from its own checks; `skipStructural` and the `*At` accessors had none. Bounded at the accessors, where every path goes through: `b` for single bytes, and s16/s32/s64, which read a multi-byte head directly and so still ran off the tail after `b` alone was fixed. The array bound was always checked -- by the JVM, and only to throw the wrong type. Sweep result across five documents, 23 690 probes of every nav entry point over every byte mutated ten ways: ~1800 untyped throws before, ZERO after. IT COSTS: decode measures +2-5% (wide-map-1000 is the worst at ~5%; skip and encode are within noise). That is a real tax on the hot path in exchange for an error contract that matches `decode`'s. Worth flagging rather than burying, since this library's positioning is performance. 2. `count` ON A SEQUENCE'S ITEMS THREW AbstractMethodError -- on ordinary, undamaged data. `Items` implemented Seqable, Indexed and IReduceInit but not Counted, so `clojure.core/count` fell through to an abstract method. Every existing test reaches for `seq`, `nth` or `reduce`, so nothing touched it. O(1) when the sequence carries an index (the sentinel node's total), a walk otherwise -- a sequence has no head to read a count from, unlike a container. Both were surfaced by harnesses that apply every entry point to every shape and mutate every byte, rather than testing each shape the way it is meant to be used. Four rounds of careful reading did not find either.
A serialization-correctness review of a1382c6 (doc/SERIALIZATION-CORRECTNESS- REVIEW.md) found eleven issues. These are the four that touch this branch or my own recent work; the rest are pre-existing and belong in their own PRs. S8 -- DEFAULT `encode-indexed` OUTPUT COULD NOT BE NAVIGATED. Its docstring says to pass the result to `boring.nav/source`; the default profile writes stringref, and nav categorically refuses a stringref document, so `(nav/source (encode-indexed v))` threw on the exact shape the docstring recommends. `:stringref false` is forced now unless the caller asks otherwise -- an index exists to be navigated, and producing one nothing can read is not a trade-off worth offering. That is the THIRD defect on this branch with the same shape: `seal-index!`'s options, `write-seq!`'s 3-arity, and now this. Each time, every test and example happened to override the default, so the advertised default was the one path never exercised. Worth treating as a pattern rather than three incidents. S1, S2, S3 -- THE SAME THREE BUGS I FIXED ON THE JVM, STILL LIVE IN CLJS: - canonical maps emitting duplicate CBOR keys (the check existed for SETS only, exactly the asymmetry the JVM had); - the canonical scratch writer not inheriting `:encode-fallback` or its re-entry guard; - map keys renewing the `:max-depth` budget, because the scratch is reset before every staged key and a reset zeroes the depth counter -- so copying the parent's depth into it did nothing. ClojureScript now carries a `depthOffset` that survives the reset and accumulates, as the JVM does, and `enter!` checks before incrementing. I fixed one runtime three times and never opened the other. That is not three misses, it is one structural gap: every test I wrote for those fixes was JVM-only, beside the JVM implementation, for a guarantee that is portable. The fix for the gap is test/boring/canonical_parity_test.cljc -- anything `:canonical` promises portably lives there now. Verified non-vacuous the way that matters: with the three writer.cljs changes reverted it produces 3 failures and 1 error on ClojureScript. And a trap found while doing it: writing a test in .cljc is NOT enough to make ClojureScript run it. bench/cljs/cljsbench/runner.cljs carries a hand-maintained namespace list whose own docstring promises "anything the JVM asserts, CLJS must assert too". The new namespace is registered, and the list is now labelled as the maintenance hazard it is.
S4 -- CANONICAL NaN HAD THREE ENCODINGS. `toHalf` carried a NaN's payload bits and sign through, so 7ff8000000000001 gave f97e00, 7ffaaaa000000000 gave f97eaa and a negative NaN gave f9fe00 -- one value, three byte sequences, under the profile whose entire purpose is that the same value gives the same bytes on every platform. ClojureScript already normalised to f97e00, so it was also a cross-platform differential. Nothing is lost by normalising: boring exposes no NaN-payload or signalling-NaN type, and decoding collapses every half NaN to Float.NaN, so those bits were never a value distinction -- only a determinism hole. RFC 8949 says a deterministic protocol without intentional NaN-payload support picks one form, and its examples use f97e00. Only on the :shortest path; :preserve-width still emits the exact f64 bits, which is what it is for. S5 -- TAG 40 ESCAPED AS RAW EXCEPTIONS. The reader cast both dimensions to Number and asked Array.getLength for the payload's length without first proving either type, so a WELL-FORMED tag 40 with wrong-shaped content threw ClassCastException or IllegalArgumentException -- contradicting doc/SECURITY.md's typed-failure guarantee. Dimensions must now be non-negative integers within array range, and the payload must be a primitive typed array, before anything is measured or allocated. The byte fuzzer rarely builds a valid tag around invalid content, which is the limitation that document already names. S6 -- THE MATRIX WRITER MISHANDLED EMPTY AND NULL-ROW MATRICES. `rowLen(rows[0])` ran BEFORE the loop's own null check, so a null FIRST row threw a raw NullPointerException and the documented null-row fallback was unreachable for exactly the row most likely to be null. And a zero-row matrix was treated as non-rectangular, taking the fallback and decoding as a PersistentVector -- losing the source type on a value with no content to disagree about. It is 0x0, it is rectangular, and its element type is known from the array's own class. Also computes the flat element count in `long` before narrowing, so a huge rectangular matrix cannot wrap past typedArrayHeader's size check. And corrects a comment that said the ragged fallback "still round-trips": it round-trips the numbers, not the type -- a ragged double[][] comes back as a vector of double[]. Declared rather than fixed, because a type-preserving frame for ragged matrices would need a private tag and RFC 8746 has nothing to say about them.
S10 -- A NEGATIVE TAG NUMBER PRODUCED MALFORMED CBOR. The registry accepted
them, and the writer's registered branch emitted through the unchecked head
path, so a handler registered as tag -1 wrote `ff` -- the CBOR break byte --
followed by its content. No exception; just output no reader can parse, from an
ordinary use of a public API.
Validated at registration, where a caller can still act on it, rather than at
emission where the value is already half written. CBOR's tag domain is
[0, 2^64-1]; this API takes a long, so it offers [0, Long.MAX_VALUE] and says
so instead of truncating.
S11 -- MMAP LEAKED ITS ARENA WHEN CONSTRUCTION FAILED. The arena owns the
mapping and the caller only learns about it through the return value, so
anything throwing after it is created leaves the mapping with no handle left to
close it. Both failure modes are ordinary rather than exotic: a missing file,
and a stringref document -- which `boring.nav` refuses by design, so the
documented "the file must have been written {:stringref false}" constraint was
itself a leak whenever a caller got it wrong. Closed on the failure path in both
`mmap-source` and `mmap-items`.
…r run
S7 -- DURATION DECODING SILENTLY RETURNED THE WRONG VALUE. Both numbers went
through `longValue()`, so `{1 1.5}` -- a valid RFC 9581 one-and-a-half second
duration -- decoded as ONE SECOND. A wrong value is the worst outcome available
to a decoder, and the writer's own `{1 seconds, -9 nanos}` subset hid it
completely, because a round trip never produces the other forms.
RFC 9581's map rules are enforced now rather than assumed: a fractional base is
carried exactly (or refused if it is not representable to nanosecond
precision), a scaled fraction requires an integer base and an unsigned value in
range, unknown unsigned keys are critical and raise, and the forms boring cannot
represent -- decimal-fraction and bigfloat bases, scaled fractions other than
-9 -- are refused with a typed error naming the key rather than reported as "no
base value" or ignored. Refusing a conforming form we cannot carry losslessly is
honest; truncating it is not.
S9 -- REGISTRATIONS FOR SCALAR TYPES SILENTLY DID NOTHING. The hottest scalars
are dispatched before the registry is consulted, so a handler registered for
`String` or `Long` never ran, while the same call for `UUID` or `URI` worked.
The same API working or not based only on which class you name, with nothing
saying which, is worse than not supporting it at all.
The lookup cannot move above these without putting a map probe in front of every
string and every long, so the registration is REFUSED -- at registration, where
a caller can still do something about it. `Writer.isRegisterableClass` keeps
that list next to the dispatch that creates it, so the two cannot drift.
Also routes the registered-writer branch through the validated `writeTag`
rather than the raw `head`, which is the emission-side half of the tag-number
check added for S10.
An Opus review agent found ten defects by mechanical stress rather than reading -- a cross-product harness over every nav entry point and shape, exhaustive single-byte mutation, and a concurrency probe. Its negative results are as useful as its findings: differential nav-vs-decode over 40 shapes x 5 profiles x 10 index settings found ZERO disagreements, mmap and heap agreed 100%, round-trips were byte-stable, writer reuse was clean. The index feature came through clean; what it found was the surface around it. WRONG ANSWERS, SILENTLY - SHARING A NAV SOURCE ACROSS THREADS. A source owns one Reader, and every cursor from it shares that Reader's mutable position and depth -- so 200 parallel passes over one `items` returned SIX plausible but wrong documents with no exception. Nothing on the surface warns you: the namespace is "read-only navigation", `Items` is a reducible that invites `fold`, and `boring.mmap` picks a shared arena precisely so the mapping is not pinned to one thread. `boring.nav/fork` gives a per-thread view. It shares the DECODED INDEX, which is the expensive part -- measured at 145 us for a 20 000-item index against 175 ns for a Reader -- and replaces only the mutable one. 200 forked passes: 200 correct. Plus a best-effort detector raising :boring/concurrent-use, which named 178 of those 200 passes. Deliberately non-volatile, so it costs nothing on the hot path, and deliberately NOT thread affinity, which would reject a legitimate handoff and push callers back to the 145 us path. It is a smoke alarm, not a lock: one pass still came back wrong without tripping it, and SECURITY.md says so rather than implying the detector makes sharing safe. - DECODE THREW A RAW ClassCastException on a corrupt `clojure/sorted-map` or `-set` frame, reachable by a SINGLE byte flip -- change one key's head to 0xF8 and an ordinary document becomes a sorted-map of simple values, which the default comparator cannot order. 8502 of 8872 untyped throwables in an exhaustive sweep. Random-byte fuzzing never builds a tag-27 frame carrying a valid name, which is why it survived every fuzz run. - `count` ON A CURSOR NEVER CHECKED ITS HEAD, the one entry point that did not. A head declaring 2^31 entries threw an untyped ArithmeticException; below that it returned an IMPOSSIBLE number -- 1048576 entries from a five-byte document -- while decode, seq, reduce and nth on the same bytes all reported :boring/bad-count. `seq` and `zipper` had the mirror bug, `(* 2 n)` overflowing to a negative long on a map head. - `build-index` STILL OVERFLOWED THE STACK on ~1.2 KB of nested containers. I made the tag chain iterative last round and left the container recursion. Bounded at 512, not the decoder's 1024, because this is a Clojure recursion whose frames give out between 600 and 800 -- a bound above the real limit is not a bound -- with a StackOverflowError conversion at the public boundary for smaller stacks. CONTRACTS THE NAMESPACE ADVERTISED AND DID NOT KEEP `(reduce f coll)` without an init threw ClassCastException (IReduceInit but not IReduce); `contains?` and `find` threw IllegalArgumentException (ILookup but not Associative); `nth`'s 2-arity returned nil where Indexed specifies a throw, turning a caller's off-by-one into a NullPointerException elsewhere; and a not-found argument was ignored through a tag, where clojure.core throws for a realised keyword or set. All of these fail on UNDAMAGED data, and all of them survived because every test used the arity the type happened to implement -- the same shape as the `Counted`/AbstractMethodError gap found a round earlier.
…order
Three published claims a reader checks by running or by cross-reading.
The four-line boring.nav introduction threw. It showed
`(nav/source bs {:stringref false})` over a `bs` the README's own encode
example three sections earlier had written with plain `boring/encode` -- which
writes stringref by default -- so run verbatim it raised
:boring/stringref-not-navigable. Worse, the `{:stringref false}` in the
snippet reads as the fix for exactly that error and is inert: nav/source forces
the option in both directions and ignores what the caller passed. The
requirement is at WRITE time, and the example now shows the write, defines its
own data, and runs: it returns "name-137". doc/STORAGE.md carried the same
snippet and the same defect.
Both pages also said "because a cursor implements ILookup, clojure.core/get-in
works on it directly", unqualified. Cursor's valAt descends MAJOR-MAP and
realises tags; an array position falls through to the not-found value. So
`(get-in (nav/source bs) ["p" 1])` is nil where `(get-in (decode bs) ["p" 1])`
is 20 -- no error, because the arity has none to give. `nth` on the array
cursor is the working form and is now printed next to the claim.
And the README said :canonical -- bytewise -- is "what fxamacker's
SortCoreDeterministic and ciborium produce". doc/COMPATIBILITY.md says the
opposite and says why: ciborium's only ordering helper, CanonicalValue, is
length-first, corrected after interop/rust/src/canonical.rs compared its output
against ours over 989 values. The correction landed in one of the two files. A
reader following the README picked :canonical to match a ciborium peer and got
a verification failure with nothing to point at. Both peers this repo actually
runs against, cbor2 and ciborium, are length-first.
Also: doc/STORAGE.md is new, referenced from README.md, three boring.core
docstrings, boring.mmap/segment-sink and SegmentSink.java, and was not in
doc/cljdoc.edn -- so seven "see doc/STORAGE.md" pointed at a page cljdoc had
never been told about. Added, with doc/IANA-REGISTRATION.md which was missing
for the same reason, and a comment at the top of cljdoc.edn saying what the
omission costs.
Verified: the README snippet run verbatim in a REPL returns "name-137";
doc/cljdoc.edn reads as EDN.
encode, encode-into!, decode and buffer -- the four most-used functions on the ClojureScript side -- were blank. On cljdoc that is what a first-time reader of the ClojureScript API sees first. buffer is the one that matters. Its JVM sibling carries the sharpest warning in the API -- "this array must not outlive the call, and must not cross a thread or async boundary" -- and the hazard is the same here minus the thread: the returned Uint8Array is the writer's live buffer, and a promise, a setTimeout, a queued WebSocket send or an IndexedDB put that outlives the call sees it rewritten by the next encode, silently. The codebase already knew this. CLJS write-seq!'s docstring says it at length, about itself, and points at "the buffer hazard is documented for buffer" -- which was the empty one. Nothing about behaviour changed. decode's text states the two platform divergences that are real and loud: shared boring.options validation, and :auto-construct-records? refused as JVM-only. bin/ci cljs -> 162 tests, 1406 assertions, 0 failures.
write-to-buffer! ends the borrow encode-buffered! records -- that fix is the
reason trim! is reachable after the documented allocation-free loop at all --
but the clearing set! is not in a finally, and the .put that raises
BufferOverflowException runs before it. So a writer that has just overflowed a
caller's ByteBuffer refuses trim! with :boring/bad-argument.
Verified: (write-to-buffer! w {:id 7 :name "hello"} (ByteBuffer/allocate 2))
throws BufferOverflowException, (trim! w) then raises :boring/bad-argument, and
one (encode-into! w {:a 1}) makes (trim! w) work again.
Documented rather than fixed: moving the set! into a finally is a behaviour
change and this pass is docstrings only. It is recoverable by anything that
encodes, which is what a caller does after a flush anyway, so it only bites
someone who flushes BY trimming.
clojure -J-Xmx6g -M:test -> 317 tests, 9534 assertions, 0 failures.
The previous commit's docstring said two runs against the same library produce byte-identical files, and offered a dirty `git status --short interop/` as evidence that the encoding had moved. Both wrong, and I had checked it with two runs that happened to agree. byte[] hashes by identity; several cases use one as a map key or set element, and PersistentHashMap orders by hash -- so two identical map literals in ONE process already enumerate their keys in different orders. The transport column is :interop, which does not sort, so the shuffle reaches the bytes. Three consecutive runs: a == b, and c differed from both in 165 of 989 rows. What is stable is the part that matters. Every difference was in column 1, the transport encoding, which each checker decodes with its own decoder before it compares anything. Columns 2 and 3 -- the canonical expectations, the actual subject -- were identical across all three runs, as a sorting profile requires. So the file's meaning does not drift and its packaging does, and a diff here proves nothing by itself. The checkers decide. Both pass against the committed copy: test_canonical_bytes.py ok, interop/rust ok (978 of 987, 9 documented divergences).
Two flaws, and neither was ClojureScript's.
THE NAME. Clojure builds a record's class name through `namespace-munge`,
which is `(.replace (str ns) \- \_)`, so `(defrecord My-Rec ...)` in
`my-test-ns` becomes the class `my_test_ns.My-Rec`. ClojureScript has no such
step: `pr-str` reports `#my-test-ns.My-Rec{...}`, the name as written. So the
BROWSER had the better name, and boring munged it DOWN to match a platform
that had lost information -- discarding what one side still had so it could
agree with the side that could not.
The munge is invertible by lookup rather than by guessing: scan loaded
namespaces for the one whose munged form is the class's package. A record
instance's namespace is loaded by construction, so the answer is always there.
`my-ns` and `my_ns` both munge to `my_ns`, so an exact match wins and a tie
falls back to the class name; `my_ns` as a Clojure namespace is vanishingly
rare. Cached per class, since the scan is O(loaded namespaces).
THE SEPARATOR. A dot cannot say where the namespace ends -- `a.b.c.D` splits
two ways and the old code guessed at the last one. A slash is legal in
neither part, so the split is exact, and it is what boring's own reserved
tag-27 names already use: `clojure/sorted-map`, `java/period`, `boring/index`.
`my-test-ns/My-Rec` on both platforms now, and identical, which it never was.
FORMAT CHANGE, sanctioned: the pre-release has only gone to an internal
channel. The golden corpus is regenerated -- 31 vectors, one byte each, `2e`
to `2f` -- which is what `golden_test`'s docstring asks for and why it asks.
Four places had to move together, which is the shape this branch keeps
finding: the wire name (`TagRegistry.recordName`), the reader's inverse
(`Reader.classNameOf`, converting `ns/Name` back to a loadable class name),
`boring.records/wire-name` for the registry key, and `auto-registry`'s
`Class/forName` probe -- which is not the wire name and had silently become
one, so the probe matched nothing and every record decoded as an
`UnknownRecord`.
`.gitignore` has said `.internal/` is not part of the published library since it was written, and three files predating that rule were still tracked. They are working notes -- audit reports and investigations -- with no bearing on anyone consuming boring, and they carry running commentary about defects in a form meant for us rather than for users. `--cached` only: the files stay on disk, they stop being published.
`:instant-type` was accepted and silently ignored on ClojureScript, against that file's stated policy -- it refuses `:auto-construct-records?` loudly for exactly this reason. JavaScript has one time type, so the JVM's `:date`/`:instant` keyword choice genuinely has no counterpart. Refusing was the other option and I nearly took it. It would break the portable `.cljc` caller who passes one options map to both platforms, which is konserve's shape and the reason this library exists. So the option takes a FUNCTION of epoch milliseconds instead. A caller using a cross-platform time library -- `cljc.java-time` or `tick`, js-joda underneath -- gets the type they want back, and boring depends on none of them. Omitted, a `js/Date` comes back exactly as before. Applied to BOTH time tags, which the first version was not: a `Date` encodes as tag 0 -- RFC 3339 text -- not tag 1, so patching only the epoch form left the option doing nothing on the shape boring actually writes. The test caught it because it round-trips a real `Date` rather than hand-built tag-1 bytes. The spec accepts a function on both platforms, so the JVM stays open to the same extension rather than having a second rule.
Three of the seven held-back findings, and all three are the same shape: a
rule that most of the code follows and one place does not.
`encode-indexed` HONOURED `{:stringref true}` alongside an index, producing a
file whose index `boring.nav` refuses outright -- a stringref is an index into
a table built from every preceding string, and a cursor holding only an offset
cannot resolve one. Its two siblings, `write-seq!` and `write-indexed!`, have
raised `:boring/incompatible-options` on that combination all along. Its
docstring argued the case for honouring it ("you simply get an index nothing
can use"), which is a strange thing to offer. Now all three refuse, and the
test asserts all three rather than the one that changed.
`:canonical` and `:canonical-order` are locked by every profile, so passing
either can only produce a conflict error. They STAY in the spec rather than
being dropped, and the docstring now says why: a dropped key is an unknown
key, so `{:cannonical true}` would become indistinguishable from a stray one.
The conflict error names the profile and the value it defines, which is more
use than silence.
And an unknown key within ONE EDIT of a real option is now refused, with the
suggestion:
:max-item -> did you mean :max-items?
:stringrefs -> :stringref
:cannonical -> :canonical
:shape -> :shapes
while `:konserve/version` and anything else unlike an option passes untouched,
which is what keeps the map open for callers who thread their own keys through
-- konserve does. Two defects on this branch were option typos, and both were
found by an audit rather than by the code refusing them. The distance check
only runs on keys the spec does not know, so the cost falls on the unusual case
rather than on every option of every call.
`register-record-class` defaulted to the raw class name, so a type registered through it went on the wire as `my_ns.My-Rec` while `encode` of the SAME TYPE wrote `my-ns/My-Rec`. Two names for one type inside one version, which is worse than either name being wrong. `TagRegistry.recordName` is the one place that decides; this now asks it. Found by an audit of the consuming projects rather than by anything here, and the reason it could hide is the second half of this commit: NOTHING in boring asserted a record's wire-name string. `register-records` -- konserve's only registration call -- `record-type-name`, `auto-registry` and `registry-for` had zero test references between them. A format change reached three consumers before anything noticed, which is what an untested public contract buys you. The test asserts the shape (namespace as written, slash, name as written), that the encoded bytes actually contain that name, and that `register-record-class` registers what `encode` writes. It runs on both platforms. The consumer breakage itself is accepted -- the serializer was never announced. What was not acceptable was boring disagreeing with itself.
Three of the four release blockers an artifact audit found. The fourth -- `register-record-class` deriving the old wire name -- was fixed in 573e7ed while that audit was writing, against the jar it had already installed. `boring.hasch` WAS NOT IN THE JAR. `src-hasch/` is an `:extra-paths` alias, so the namespace this CHANGELOG advertises as a feature was absent from every released artifact, and konserve and datahike are its named consumers. The failure is silent rather than a missing-namespace error: with hasch on the classpath and boring's integration missing, two different record types and a plain map all content-address to the SAME uuid. It loads only when hasch is present, which is what the namespace is built for, so shipping it costs a consumer without hasch nothing. Three shipped docs asserted that boring's reserved tag-27 names "carry a slash, which a JVM class name never does, so a user record can never collide". Every record name now carries a slash, so that invariant is gone. It is removed rather than left standing, and replaced with the weaker statement that is still true: colliding needs a record named `sorted-map` in a namespace named `clojure`, and a caller's registry is consulted before the built-in markers in both directions. `EXTENDING.md`'s registration recipe used the old dotted name and did not fire if followed verbatim; corrected, and run. The CHANGELOG now carries the wire-format break with the two-line re-registration a caller with existing data needs, and the reasoning: the JVM had LOST the namespace as written while ClojureScript still had it, and boring munged ClojureScript down to match the platform that had lost it. Plus the five other user-visible changes that were undocumented -- the one-edit option typo refusal, `Items.nth` throwing, ClojureScript `write-seq!` forcing stringref off, `:instant-type` taking a function, and `:trust-index :ignore`. 0.1.4 through 0.1.10 are on Clojars. The break is accepted deliberately -- boring is unannounced and datahike has not shipped against it -- but it is accepted in writing rather than by omission.
Two guards that could not fire, and one invariant that quietly stopped holding. `clojure -M:format` exited 1 on six committed files, so the CI format job was permanently red -- and a job that is always red is a job nobody reads. The cause was not drift: `dev/gen_golden.clj` emitted BOTH frozen maps at three spaces, while `jvm-only` sits a level deeper inside a reader conditional, so regenerating the corpus un-formatted the file every time. The generator now takes an indent, and regeneration leaves the formatter satisfied. Verified by regenerating and re-checking, not by formatting once. `bin/check-artifact` was invoked without `--release`, so its snapshot guard -- written to refuse publishing a SNAPSHOT -- could never fire on the one path that publishes. The flag also needs `BORING_VERSION`, which CI does not set since the version is computed from the revision count, so it is passed explicitly rather than hoped for. Verified both ways: a snapshot version is now refused, a real one passes. AND SHIPPING `boring.hasch` BROKE TWO THINGS THAT SHIPPING IT REVEALED. `boring.core`'s optional-integration probe caught "the integration namespace is absent". That was the same condition as "hasch is absent" only while `boring/hasch.cljc` was missing from the jar -- which it silently was, in every release. With it present the require FINDS the namespace and fails inside it on `hasch.benc`, wrapped in a Compiler$CompilerException neither catch matched, so `(require 'boring.core)` threw for every consumer without hasch. It probes `hasch.benc` directly now, which is what the comment above it always claimed. And a record hashed WITH its class present stopped matching the same record hashed without it -- the exact invariant `boring.hasch` exists to hold. hasch coerces a live record through its class name and incognito's writer does `(-> r type pr-str normalize-ns symbol)`; both land on the munged dotted form, while boring's wire name is now the true `namespace/Name`. The wire name and the hash name are different things, and the bridge translates between them -- `incognito.base/normalize-ns` reimplemented rather than depended on, since boring's only runtime dependency is Clojure.
`bit-shift-left` is 32-BIT on ClojureScript. The shift form is correct on the
JVM and truncated the frame's 8-byte back-pointer there -- BEFORE the range
test whose own comment says it "rejects a nonsense pointer" ever saw the value.
Measured before the fix, over an exhaustive single-byte sweep of a sealed
file: 20 cases where both platforms decoded successfully and DISAGREED about
how many items the file holds. One file, two logical contents, no error on
either side. After: the same sweep gives an identical distribution on both --
{27 52, :err 234, 25 171, 26 46, 32 2, 33 1} across 512 mutations.
The two sides genuinely need different arithmetic, so `be64` says so in a
reader conditional rather than picking one and hoping. Multiplication is exact
on ClojureScript to 2^53, far past any file length; on the JVM it is CHECKED,
and a pointer with the high bit set would raise a raw ArithmeticException out
of the function whose whole job is deciding whether to trust these bytes --
which is why that side keeps the shifts. A value above 2^53 fails the range
test as the nonsense pointer it is.
Both readers of those 8 bytes now go through it. `index-frame?` had its own
copy of the loop, which is the shape that produced this in the first place.
`(= cursor x)` threw `java.lang.AbstractMethodError` on UNDAMAGED data.
`clojure.lang.Associative` extends `IPersistentCollection`, so declaring it
obliges `equiv`, `cons` and `empty` as well as the `count` and `seq` that were
implemented. And it is an `Error`, so a caller's `catch Exception` does not see
it. Third instance of that family in this file -- `count` threw the same way
before `Counted` was added, `reduce` before `IReduce`. Declaring an interface
is a promise about every method on it, including the inherited ones.
`equiv` is IDENTITY. Realising a cursor to answer `=` would do arbitrary decode
work behind an operation that reads as free, which is the same argument that
keeps `Cursor` out of `IDeref`.
ClojureScript's `seal-index!` took the stride from its own options where the
JVM takes it from the index map, so the documented `build-index` +
`seal-index!` pair sealed a frame claiming one stride over anchors laid at
another -- 8 of 9 combinations. The index is then silently dead: `nav` jumps by
the claimed stride and walks 39 items where 3 would do, with no error.
`:instant-type` as a FUNCTION was honoured on ClojureScript only, though the
shared spec accepts it on both -- so a portable caller passing one options map
got their type in a browser and a `java.util.Date` on the server. That was mine,
from the commit that introduced the function form. Two attempts: the first used
bare `ifn?`, which a KEYWORD satisfies, so `:date` and `:instant` were invoked
as constructors and every instant decoded to nil.
And ClojureScript's structural skip treated info-31 as indefinite for EVERY
major type, where only 2-5 have an indefinite form. Both platforms still refuse
`1f` and `3f`, so nothing was accepted that should not be -- they disagreed
about why, `:boring/reserved-info` against `:boring/truncated-input`.
Verified rather than assumed: an exhaustive two-value sweep over every byte of
a sealed file now gives an identical outcome distribution on both platforms,
and the empty-map depth asymmetry the sweep also reported does not reproduce --
`[[]]` and `[{}]` both raise at `:max-depth 1` on the JVM.
`boring.frame/ends-at` names ONE rule and binds it to `Reader.skipFrom` on the
JVM and `boring.reader/skip-from` here. They did not implement the same rule,
and this one is the inner loop of `build-index` -- so a browser built an index
over documents neither platform can decode, and `boring.nav` on the JVM is what
then opens the file.
Executed at HEAD, `skip-from` against `Reader.skipFrom`, JVM verdict first:
83 ff 01 ff 02 03 break inside a DEFINITE array unexpected-break / reserved-info
ff a bare break unexpected-break / reserved-info
bf 01 ff indef map, break mid-pair unexpected-break / :ok
5f 20 ff indef BYTE string, int chunk bad-indefinite-chunk / :ok
7f 41 61 ff indef TEXT string, bytes chunk bad-indefinite-chunk / :ok
2000 nested 81 nesting past the skip bound max-depth-exceeded / :ok
2000-long c0 chain tag chain past the skip bound max-depth-exceeded / :ok
20000 open 9f " max-depth-exceeded / UNTYPED
The last one is the one doc/SECURITY.md forbids outright: a RangeError with
empty ex-data out of a public read path, on the one platform browsers run.
`core.cljs`'s own INDEX-WALK-MAX-DEPTH does not cover it, because indefinite
containers are delegated to `skip-from`.
The walk is now EXPLICITLY ITERATIVE over a stack of open containers rather
than a flat "items owed" counter. The counter is why there was no bound to
apply: 2000 nested `81` never raises it above 1, so nesting was invisible to
it. Depth is a number that is checked now, not a stack that runs out --
`(dec (.-length stack))`, against the same `max(maxDepth, 1024)` the JVM uses,
and the tag chain bounded by its own length as `skipStructural` does.
The `-3` frame is the one that is not obvious: an indefinite MAP owing a value
cannot be closed by a break, which is what makes `bf 01 ff` refuse. A single
"owed items" count cannot express that and silently accepted it.
Empty containers mirror the JVM's asymmetry deliberately -- an empty array
costs no level, an empty map costs one -- because two walkers agreeing is worth
more than either being tidy, and it is only observable at the bound.
AND THREE MORE OF THE SAME SHAPE, all measured against the JVM:
The positional accessors used a bare `aget`. Past the end a Uint8Array yields
`undefined` and `(bit-shift-right undefined 5)` is `0`, so offset 99 of a
two-byte buffer read back as an unsigned-integer head of length 0 instead of
raising. `major-at`/`head-arg-at`/`head-end-at` are public and `index-walk*`
calls them directly. `Reader.b(long)` already records this as a fixed JVM
defect -- 415 of 5360 mutations of an unindexed document -- and says it is the
navigator's contract with damaged data. The port never got it.
`read-map!` skipped `enter!` for an empty map where the JVM charges a level, so
`[{}]` decoded at `:max-depth 1` in a browser and raised on the server. That is
a security bound doc/SECURITY.md tells operators to TIGHTEN, so tightening it
made a portable pipeline reject its own valid documents.
`encode-indexed` overwrote `:stringref` with false before any gate: an explicit
`:stringref true` is `:boring/incompatible-options` on the JVM and was silently
honoured-as-false here, and a garbage `{:stringref "yes"}` was thrown away
unread where the JVM says `:boring/bad-option`. Options are validated first
now. `seq_index_test.clj:441` is named "refused by every writer" and ends
"Three functions, one rule, and one of them did not follow it"; there were
four, and the test is .clj so it could not see the fourth.
Verified: test/boring/skip_parity_test.cljc is .cljc, runs on BOTH platforms,
and carries the JVM's verdicts as the expected values. Against this tree it is
7 tests / 77 assertions green on both. Against the unfixed tree it is 30
FAILURES on ClojureScript and 0 on the JVM -- which is the shape a parity test
has to have to mean anything. Registered in cljsbench.runner, because a .cljc
test that nobody adds to that hand-maintained list covers one platform.
Gates: JVM 327/9631 0 failures, cljs 172/1500 0 failures, lint, hasch,
fuzz-cljs (50000), cljfmt.
boring builds an index two ways: the writer captures nodes while encoding (`write-indexed!`, `write-seq!`) and a byte walk derives them afterwards (`encode-indexed`, `build-index`). `the-two-index-builders-agree` asserts byte identity of the whole sealed file -- over ONE profile, ONE `:index-min`, and four values, none of them tag-27 wrapped. Both places the builders actually disagree are outside that box, so the test could not fail for either. THE `sorted` FLAG UNDER `:profile :canonical-rfc7049`. That profile sorts keys LENGTH FIRST, so `writeCanonicalMap` took `sorted = !legacyCanonicalOrder` and claimed nothing at all, while the byte walk compared the emitted key bytes -- which is what `Reader.compareItemsAt`, the comparator the navigator's binary search actually uses, compares -- and reported the truth. Executed, a 20-key nested map at stride 4 and 16: same bytes, same offsets, `sorted [true false ...]` from the walk against `[false false ...]` from the writer. The conservative answer is SAFE: it only ever gives up a binary search. It gave one up for every `:canonical-rfc7049` file written through `write-seq!`, including the many whose keys are bytewise ascending anyway -- most of them, since length-first and bytewise agree unless key lengths differ. It is read off the emitted key bytes now, in the loop that was already comparing every adjacent pair to reject two keys that encode identically. No new comparisons. A TAG-27 FRAME'S OWN `[name, args]` ARRAY. For a `sorted-map` or `sorted-set` the byte walk emitted a node for the wrapper as well as for the collection inside it: containers `[2 22]` against `[22]`, counts `[2 40]` against `[40]`, 306 bytes against 295. `boring.nav` never descends a tag structurally, so that node can never be consulted -- it was 11 bytes of dead weight and a false byte-identity claim, not a wrong answer. Dropped on the walk side, on both platforms, because the walkers must also agree with each other. It needs `:index-min` <= 2 to appear at all, which is why the default of 16 hid it. Verified: the widened sweep is a NEW test rather than an edit of the old one, so the original stays the regression it was written to be. 6 values x 5 profiles x 3 strides x 2 `:index-min` = 180 pairs, compared as decoded node structure AND as bytes, with a third assertion that both files still read back as the plain encoding -- so a builder cannot be made to agree by making both wrong. The specimens include keys where length-first and bytewise genuinely diverge (short text against large integers), which equal-length keys cannot show, and three tag-27 shapes. Against the unfixed tree: 120 failures, and it names the flag and the node count. Against this tree: 49 tests / 7203 assertions, 0 failures. Gates: JVM 328/10171 0 failures, cljs 172/1500 0 failures, lint, hasch, cljfmt.
The class-to-name cache I added with the wire-name change made the encoding LOAD-ORDER DEPENDENT, permanently. The lookup inverts Clojure's namespace munge by scanning loaded namespaces, so its answer depends on what was loaded when it first ran -- and caching the fallback froze that: an audit measured the same record encoding as `wn-ns.core/R` in one JVM and `wn_ns.core/R` in another, with reloading the namespace not fixing it. Wire bytes must not depend on class-load order. A fallback is "not resolved yet" rather than an answer, so it is recomputed until the namespace is present; once resolved the name cannot change, so caching from then on is sound. The test for a resolved name is that no `_` survives, which costs a genuinely underscored namespace its cache entry and nothing else. That the cache existed at all was a performance answer to an O(loaded namespaces) scan. It is still that -- for every name it can actually resolve.
`Writer.java`'s `plainMap` excludes records, sorted maps and metadata-carrying
maps from a shaped array, with the comment that is the whole rule: "an
optimisation that silently changes the value is not one." `writer.cljs`'s
`homogeneous-shape` -- the second implementation of that same rule -- checked
`record?` alone.
Three things were destroyed on ClojureScript and not on the JVM, all silently,
since the bytes are valid CBOR and the loss surfaces as a changed value much
later:
- a sorted map came back unsorted
- metadata was dropped while `:incl-metadata?` was still true
- an `UnknownRecord` was flattened -- it is `map?` and NOT `record?` here --
breaking the documented guarantee that it re-encodes to identical bytes
Verified on ClojureScript after the fix: all three keep their sortedness,
metadata and type, and a vector of plain maps still shapes, so the guard
excludes what it should and nothing more.
Found by a differential sweep over ~58,000 compared cells. It is the same shape
this branch keeps producing -- one rule, two implementations, one of them
weaker -- and the JVM comment explaining WHY the guard exists had been sitting
next to the correct copy the whole time.
Nine tags where the JVM and ClojureScript implement the same rule differently. Seven close here; the eighth was fixed in the same working tree and swept into fc6c77b by a concurrent commit; the ninth cannot close and is now documented instead of denied. WHERE THE JVM WAS THE WRONG SIDE: Tag 39649's shape keys must be DISTINCT, and that fell out of building each row's map -- so it inherited two properties that do not belong to it. With ZERO rows nothing was built and nothing was checked: `d99ae1 82 82 0101 80` decoded to `[]` here and `:boring/bad-tag-content` on ClojureScript. And it was gated on `:check-duplicate-keys`, which is an option about MAP CONTENT, so with the option off `39649([[1,1],[[1,2]]])` was `[{1 2}]` here and refused there. Repeated keys make the shape itself meaningless -- the row values have nowhere distinct to land -- so it is a property of the tag content and no decode option may turn it off. Checked once, where the keys are read, with the same two-tier strategy as `checkDistinct` so the work is never attacker-controlled O(n^2). (Tag 1002 used the fraction VALUE as its "fraction seen" flag, so a CBOR null degraded the key to absent -- `{1: 5, -9: null}` was PT5S -- and with the key unregistered RFC 9581 3.3's "at most one scaled fraction" could never fire, so `{1: 5, -9: null, -3: 1}` was PT5.001S. Both are refused now. That edit is already in the tree under fc6c77b, which committed my uncommitted working file along with its own change; recorded here because the commit message there does not mention it.) WHERE CLOJURESCRIPT WAS: A tag-4 exponent is range-checked against the 32-bit scale a BigDecimal can carry. `c4 82 1a80000000 01` -- exponent 2^31 -- decoded to a Decimal no JVM peer can construct, so a document written in a browser had no reading on the server. The mantissa is deliberately not bounded; a bignum mantissa is what tag 4 is for. Tag 30 is READ THROUGH `Numbers.divide` on the JVM, which reduces to lowest terms, puts the sign on the numerator, and yields an integer when the denominator reduces to 1. This platform kept whatever was on the wire, so `30([4,2])` re-encoded as itself instead of as an integer and `30([1,-2])` and `30([-1,2])` were two different values. Measured as BYTES, because the two platforms genuinely have different types here -- Ratio against Rational -- and the bytes are what crosses. The integer case deliberately stays a BigInt: that is what makes it re-encode `c24102` as the JVM does, where narrowing it would have traded one round-trip divergence for another. Tag 32's grammar approximation had one hole left. `java.net.URI` requires a non-empty scheme-specific part, so `"a:"` and `"urn:"` are refused there and were accepted here. The part ends at a `#`, not at the end of the string -- `"a:?q"` is accepted and `"a:#f"` is not -- which was measured across sixteen forms rather than read off the class's javadoc. WHAT CANNOT CLOSE: tag 35. `java.util.regex.Pattern` and `js/RegExp` accept different pattern SOURCES, so `35("(?i)abc")` compiles on the JVM and is `:boring/bad-tag-content` in a browser. Translating regex dialects is not a codec's job. doc/COMPATIBILITY.md said "a regex is symmetric, so tag 35 has no caveat", which is true of the type and false of the language; it now says what actually differs and what a portable pattern has to avoid. Verified: the cases go into test/boring/skip_parity_test.cljc, which is .cljc and runs on both platforms, with the JVM's answers as the expected values except where the JVM moved. Against the unfixed tree: 4 failures on the JVM (2 more with the tag-1002 flag reverted by hand, since that edit is committed), 5 on ClojureScript, naming the tag and the bytes each time. Against this tree both are green. Gates: JVM 331/10193 0 failures, cljs 175/1522 0 failures, lint, hasch, fuzz-cljs (50000), cljfmt.
`boring/unencodable` is the `:encode-fallback :placeholder` value -- a tag-27
frame naming the type that could not be encoded. On ClojureScript it named the
type with `(str (type x))`, which is the constructor's COMPILED SOURCE: for
`(atom 1)` the placeholder carried
`function He(){this.state=1;this.meta=...;...}`.
Three things wrong with that, and the third is the one that matters. It is
unbounded -- the placeholder grows with the size of the constructor. It leaks
implementation into a value that gets WRITTEN TO STORAGE. And it differs in
every build, which defeats `:archival`, whose entire purpose is that two dumps
of the same value compare equal.
The constructor's `.name` is minified under `:advanced` and so is not stable
across builds either, but it is SHORT and carries no source. There is no stable
type name on this platform -- that is the same constraint that makes
`record-type-name` read `pr-str` rather than the constructor -- so the
placeholder now says what it can instead of pasting a function body into the
wire. Measured: 2 characters where it was a full function definition.
mmap was checked and needs no change: `mmap-source` and `mmap-items` already
return `[cursor arena]`, both docstrings say the caller closes it, and the
arena is closed if construction throws. An arena dropped without being closed
does leak the mapping, but a `Cleaner` safety net would close it when the
returned VECTOR became unreachable while a derived cursor could still be live
-- turning a leak into a use-after-free. Explicit ownership is the right
answer, and it is what konserve-lmdb does with `mdb-env-close`.
`{1: true, 2(h'01'): false}` decoded as a two-entry map in a browser and raised
`:boring/duplicate-map-key` on the server. That is a parser differential in the
direction `doc/COMPATIBILITY.md` does not consider -- a document a BROWSER
accepts and the JVM refuses -- on exactly the boundary kabel peers sit either
side of.
Both platforms decode `c2 41 01` to a bignum, measured. What differed was the
equality the duplicate check uses: `(= 1 1N)` is true in Clojure and
`(= 1 (js/BigInt 1))` is false in ClojureScript. So this is a CHOICE, unlike
the two rows either side of it in the report -- `1` versus `1.0`, and `0`
versus `-0.0` -- which JavaScript genuinely cannot represent apart, making
"duplicate" forced rather than chosen there.
Chosen to match the JVM because boring's own canonical rule reduces a bignum
that fits to a basic integer: treating them as one key is what the rest of the
codec already believes about them.
The test asserts the refusal AND two controls -- a bignum too large to equal
the integer key stays a second key, and two plainly different integers are
untouched -- because a check that refused every bignum-plus-integer map would
pass the first assertion on its own.
Seven reserved markers decoded to their bare payload here, so re-encoding emitted a plain text string or array and the frame was gone. Measured over the frames the JVM writer actually emits: clojure/char, java/period, java/char-array, java/boolean-array, java/string-array and java/object-array all came back as different bytes than they went in, while the four whose type ClojureScript HAS -- queue, sorted-map, sorted-set, ex-info -- were identical. A JVM peer received a String where it had sent a java.time.Period, and a relay downgraded every such value it passed through, permanently. That is the failure doc/COMPATIBILITY.md refuses for URI, against the promise two paragraphs above it: "a round trip through ClojureScript never corrupts a document; it just hands back less than the JVM would." The fix is to DELETE the special cases, not to add machinery. Each marker keeps its validation exactly as it was -- accepting what the other platform refuses is a parser differential whatever value this side builds -- and then hands back through the same `frame-fallback` the unregistered-name default already used. Both writers already encode a TaggedLiteral to its frame, so `(encode (tagged-literal 'clojure/char "a"))` produced the JVM's bytes on either platform before this commit; only the receive half was missing, which is why the send direction needs nothing new and is now pinned by a test. java/period additionally accepts only the CANONICAL form now, on both platforms. `Period.parse` takes far more than `Period.toString()` emits -- lower case, a leading sign, per-component signs, weeks, leading zeros -- and tracking that by hand had gone wrong in BOTH directions: `p1d` was accepted on the JVM and refused on ClojureScript, and `P2147483648D` was accepted on ClojureScript and refused on the JVM, a browser admitting a document its server rejects. Widening could not fix it either, because a Period holds years, months and days and no spelling: 8 of 25 legal inputs re-encoded as different bytes on the JVM (P1W -> P7D, -P1D -> P-1D, P1Y0M -> P1Y, P1Y1W1D -> P1Y8D, ...), and boring keys content by bytes. Accepting only what can be faithfully stored makes the two platforms agree AND every accepted document byte-stable. The JVM decides it definitionally -- parse, then require the result to print back to the input -- and ClojureScript spells the same rule as a regex, with `period-domains-agree` holding the two level over 25 inputs. Nothing legitimate is refused: boring's writer emits Period.toString(), and java/period is boring's own reserved name, so no other producer writes it. Same trade as refusing RFC 3339 offsets past +/-18:00. The round-trip test also runs under :shapes, :canonical, :archival and :interop, because the array-payload markers now hand back a vector and a homogeneous vector is what the shaped-array writer looks for. Measured clean on both platforms.
`register-record "taoensso.nippy.StressRecord"` stopped matching when a
record's wire name became its true `namespace/Name` -- the breaking change
CHANGELOG.md documents, with this exact migration as the example. The
registration was never updated, so `bin/ci`'s nippy-stress stage had been red.
Worth noting how it failed, because it is what a consumer who misses the
migration will see: no error, no unregistered-name warning, just a
StressRecord coming back as `#boring/record ["taoensso.nippy/StressRecord"
{:x "data"}]`. The stress test caught it only because it asserts equality with
the input.
Both are breaking for a ClojureScript caller, and the CHANGELOG is where someone finds that out. COMPATIBILITY.md gains the per-marker table it was missing -- the tag table two sections down already carries the equivalent for URI, Duration, LocalDate, Decimal and Rational, and the tag-27 markers were the one group with no such row while being the group that had silently lost the property those rows promise. Also records the send direction, which is the part that is easy to miss: both writers encode a TaggedLiteral to its frame, so a browser can originate a `char[]` or a `Period` it has no type for.
Every `bin/ci` run rewrote interop/canonical_fixture.cbor with different bytes
and left the tree dirty, so the file the header calls committed-and-current
could never converge, and a stale commit was indistinguishable from noise.
Two causes, both the same mechanism: a byte array hashes by IDENTITY, its slot
in a hash-ordered collection moves on every JVM run, and boring writes a
collection in iteration order.
- `order-mixed-majors` was a 9-entry map literal -- one past the array-map
threshold, so a PersistentHashMap -- with a byte-array key.
- `gen-cases` built its 500 generated maps with `zipmap` and its sets with
`set`, sampling from a `key-pool` holding five byte arrays.
Maps become `array-map`, which keeps insertion order at any size, at no cost
to what is tested: the canonical expectations sort their keys, which is the
property these rows exist for. Sets have no insertion-ordered counterpart and
ANY identity-hashed member destabilises them, so byte arrays are filtered out
of generated sets -- the one place this costs coverage, kept by the new
singleton row `order-set-singleton-bytes`, where there is no order to vary.
The filter is applied after sampling so the RNG draw sequence is untouched and
the other generated cases are the values they already were.
Not an encoder fault anywhere in this: the maps' own iteration order varied
and boring wrote it faithfully.
The generator now REFUSES to write a corpus that would not reproduce, naming
the offending cases, and `bin/ci` compares the regenerated fixture against the
committed one. Both were needed. Repeated runs are not evidence: HotSpot's
identity hash is a per-run PRNG that repeats for the same allocation order, so
three consecutive regenerations agreed while this was still broken and only
diverged under `bin/ci`. Removing the set filter and re-running confirms the
assertion fires rather than decorating -- it names gen-set-6, gen-set-26 and
gen-set-30.
The `bin/ci` half additionally catches the other failure that hid in this gap,
which nothing checked: a committed fixture gone stale against the encoder. The
checkers cannot see either, because they read the file just written.
An unregistered tag-27 name decodes to an UnknownRecord, which is lossless passthrough and the whole reason the carrier exists -- a relay must be able to carry a type it has no constructor for. The cost is that a registration which can NEVER match looks exactly like data that legitimately has no constructor. That is how the record wire-name change went unnoticed in boring's own nippy suite: the registry was keyed on the munged class name, the wire carried `namespace/Name`, and every StressRecord came back as an UnknownRecord with no error and no warning. It was caught only because that suite asserts equality with the input. :fallback the default, unchanged :error :boring/unregistered-record (fn [name payload]) return value is used No `:warn`. boring would have to own an output channel (`*err*` versus `js/console.warn`) and dedupe state to keep a 200 000-item log from flooding, and neither is testable without capturing output. The function form is that capability without boring picking a logger for its consumers, and it also covers substitution. `boring.data/frame-for` is now public -- the rule both readers use to choose a carrier -- so a handler that only wants to warn returns the default rather than reimplementing it and drifting. THE KEYWORDS ARE CHECKED BEFORE CALLABILITY, on both platforms, because a Clojure keyword is `ifn?`: a reader that tested callability first would invoke `:fallback` as a function of [name payload]. That exact confusion already shipped on this branch -- `:date` and `:instant` were invoked as instant constructors and every instant decoded to nil -- and it is silent, since invoking a keyword is legal. The option matrix pins all three legal forms as accepted so a side that forgot the option fails, and both illegal ones as :boring/bad-option so neither side widens to "anything goes". RESERVED MARKERS DO NOT TRIP IT. `clojure/char` and `java/period` are known names even where the type behind them is missing, and on ClojureScript they now reach the very same fallback helper an unregistered name does -- which is exactly how they would have. Asserted on both platforms. Also closes a gate hole this work exposed: `run` records a failure and carries on, so a failed cljs-compile left the previous run's target/cljs-test.js in place and cljs-tests ran THAT, reporting ok with the old test count against code that does not compile. Observed live -- a broken docstring failed cljs-compile while cljs-tests passed 178 stale tests, and fuzz-cljs did the same. Both stages now delete the artifact first, which also catches a compile that succeeds and writes nothing. Verified by breaking reader.cljs on purpose: cljs-tests goes red instead of green.
doc/EXTENDING.md explained both registration mechanisms and never said what
the fallback is FOR, so the default that makes boring usable in a p2p system
read like a consolation prize. The argument it was missing: the type name
travels in the data as tag 27, so a peer with no classpath entry for a type
still holds the name and the fields, can store and index and re-encode it
byte-identically, and the far end reconstructs the record. The middle peer
participates without being coupled to types that are none of its business.
This is what incognito provides for fressian; boring needs no companion
library because the name is in the format.
Also answers the question that decides whether that story actually holds --
does manipulating one as a map keep the type -- with a measured table rather
than a claim, and pins it with `unknown-record-keeps-its-type-through-map-
operations` so the table cannot rot. assoc/assoc-in/update/update-in, dissoc
(including of every key), conj, merge onto, into, empty and with-meta all stay
an UnknownRecord and re-encode to their tag-27 frame, identically on both
platforms. select-keys, `into {}` and merging INTO a map build a fresh map and
lose it -- which is Clojure's boundary rather than boring's: measured side by
side, a real defrecord loses its type at exactly those three, and
UnknownRecord is more forgiving at two others (dissoc of a basis field
degrades a defrecord to a map, and `empty` on a defrecord THROWS). The rule
worth carrying is that if an operation would keep a defrecord a record, it
keeps an UnknownRecord an UnknownRecord.
The ClojureScript equality asymmetry was documented only in a source comment,
where a user deciding whether to rely on `=` will not find it.
Corrects a stale paragraph while here: it still said the JVM writes the class
name and ClojureScript munges to match, which is the behaviour the wire-name
change removed and the opposite of what now happens.
And states the registration guidance that would have prevented the nippy
breakage: `auto-registry` and `register-record-class` derive the name from the
type, so a change to the naming rule cannot leave them behind, while a
hand-written string silently stops matching. Where a hand-written name is
unavoidable, `:on-unknown-record :error` turns the silence into a typed
failure.
Three things this branch shipped that never executed. `bench/cljs/cljsbench/skipprop.cljs` asserts the property the whole index feature rests on: `skip-from` must land exactly where `read!` lands, because a skip one byte off yields an offset into the middle of an item, which reads back as a plausible WRONG VALUE rather than an error. It was not a `bin/ci` stage, not a compile target, and not required by the runner. Now part of stage_fuzz_cljs -- it is randomised, 20 000 generated values per run -- which needs no CircleCI change, since the `fuzz` job already calls `bin/ci fuzz-cljs`. IT COULD NOT HAVE FAILED IF IT HAD RUN. `(set! (.-exitCode js/process) ...)` compiles, runs, and does nothing under `-O advanced`: `process` is not in this build's externs, so Closure renames the property and node never sees an exitCode. Measured by inverting the property -- 20 000 of 20 000 disagreements, reported on stdout, exit code 0. `bin/ci`'s `run` greps its logs precisely because a zero exit is not sufficient, and the old wording matched none of its patterns either, so the gate would have passed both ways at once. Fixed with `goog.object/set` and a string property name, which Closure does not rename, plus a `FAIL:` prefix the grep recognises. Verified by inverting again: exit 1 and a matching line. `bench/cljs/cljsbench/runner.cljs` -- the whole ClojureScript suite -- had the same broken exitCode, and the evidence was already in CI output: it reported `FAILED (non-zero failure count)` rather than `FAILED (exit 1)`, and that branch is only taken when the command SUCCEEDED and the log had to be grepped. The grep is why nothing broke; it is not why it should stay wrong, and anyone running `node target/cljs-test.js` outside `bin/ci` got a green shell on a red suite. `bin/index-mutants` is the gate that proves the index tests DEPEND on the index -- it damages the index in known ways and requires the suite to go red for each. `bin/ci` leaves it out of `all` on purpose, with a comment saying it "belongs in CI proper rather than in a local bin/ci run", and it was never added to .circleci/config.yml. So it belonged nowhere and had run nowhere. Now its own job, like fuzz, because it runs the index suite four times over. First execution passes and is meaningful: index load 9 tests, Items.nth anchors 1, lookup-map 3, nth-item 2. Deleted `bench/perf_worktree.clj` and `bench/cljs/cljsbench/perf_worktree.cljs`. Their docstrings call them probes for the performance worktree; they are named after `.internal/perf-worktree`, which is gitignored and not in the repo, and nothing references them. `.internal/BRANCH-FIXED-13.md` flagged all of the above and could not act -- deleting tracked files was outside the paths that audit was given.
`encode` regressed 3.9-5.2x on small payloads on this branch, and the cause was
not in the encoder: the Writer's index arrays were FIELD INITIALISERS, so every
Writer paid for them, including the ones `encode` builds -- which can never
index, because `encode` emits one CBOR item and refuses to append a frame by
design.
private long[] idxSeq = new long[1024]; // 8 KB, every writer
private long[] idxOffs = new long[32]; // + three more
Measured against the merge base on one machine, back to back, same script:
main branch fixed
(writer 256) 0.060 us 1.437 us 0.068 us 24x, then gone
small-map encode 0.420 1.630 0.371
mixed encode 0.351 1.817 0.384
datom-200 encode 82.7 83.6 79.5 never visible here
Writer allocation 512 B 9 408 B 592 B
The regression hid because a fixed ~1.4 us disappears into an 83 us datom
encode -- it is only legible on the payloads `encode` is actually used for, and
`encode` is the primary API. Allocation is the clearer statement of it: 9 408
bytes of garbage per call where 512 had done.
Allocated lazily now at the two points that grow them, `reserveNode` and
`idxItem`, both of which already run only while indexing, with the five public
accessors returning empty rather than NPE-ing for a writer that never indexed.
`trim()` now RELEASES the arrays instead of reallocating them at the default
size. It exists to give memory back, and since the arrays are lazy, dropping
them returns more and costs nothing until the next index is built.
Verified beyond the suite: write-seq! output is byte-identical across a
trim-and-re-index cycle, nav still reaches item 4999 of 5000 through the index,
every accessor answers 0 on a fresh writer, and bin/ci mutants reports the same
four index paths with the same dependent-test counts as before the change.
Found by re-running the published benchmarks before merge. Note they cannot be
reproduced as published on a machine with the CPU governor in powersave: every
codec measured ~3.3x slow, uniformly, boring and hako and nippy alike. What
made this finding possible was the RELATIVE gap -- boring's small encodes moved
13x where hako's moved 3x -- and then an A/B against the merge base on the same
machine in the same state, which is the only comparison that state cannot
distort. Wire sizes are deterministic and came back byte-identical to the
published tables, all six payloads, raw and compressed.
…pport Two unrelated things the benchmark re-run turned up. interop-canonical died on CircleCI with `cannot import name _decoder from cbor2` while passing locally. cbor2 5.5 renamed its pure-Python modules to _decoder/_encoder/_types; the workstation had 5.8.0 and CI's image resolved something older, because the install was unpinned. The script accepts both layouts now -- they are the same code under two names -- and the CI install is floored at 5.5 so the C-extension/pure-Python PAIR this gate compares is a known one rather than whatever the image happens to resolve. The comment above that install already warned a version difference had broken tag 39 here once. Verified by forcing the legacy branch: same verdict, 986 values, ok. Then the claims. Re-running `clojure -M:bench -m published` and `clojure -M:nippy-bench` on a quiet machine at the performance profile reproduced both documents closely -- every wire size byte-identical, the nippy-bench table within 8%, boring's own column with no cell off by more than 15%, and README's 33x typed-array claim landing at 33.3x. What did not hold: - "boring beats nippy on all twelve timing cells", in both README.md and doc/PERFORMANCE.md, is refuted by the table printed directly above it: small-map decode reads 0.31 against nippy's 0.29, and long-vec-1k decode 11.05 against 10.23. Nippy has always won two cells there. The re-measurement put it at four, adding mixed and nested-map-50 decode. Replaced with what both runs support: every encode cell, plus the two large map payloads on decode (2.5x and 1.6x), with the small decode cells trading places run to run. - "1.6x nippy/fast" and "4.8x fressian" measured 1.50x and 4.16x this time. Quoted as ranges, with the weaker bound stated, since fressian is the one that moves. - "boring+zstd matches nippy's round-trip time" measured 1 186 against 1 099. Now stated as within 10%, with both measurements given. The tables themselves are left alone: boring's column matches what is published, which is also evidence that the published numbers predate the Writer allocation regression fixed in the previous commit and that the fix restored them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A CBOR sequence has no way to reach item n except by skipping the n-1 before it, and a map has no way to reach a key except by stepping over the values in front of it. Fine while tailing a log, useless while seeking — reaching the last of 200 000 items measures 10.6 ms by skipping and 0.2 µs through an index.
Reproduce everything here with
clojure -M:bench -m nav index.Layout
Tag 27 with a name, not a number of its own. The index occurs once per file, so the 14-byte name costs 0.05% — against which it buys one IANA registration instead of two, self-description (
cbor2sees the string, not an unregistered number), and a strictly narrower false-positive target. Tag 39649 keeps a bare number for the mirror-image reason: shaped arrays occur per array, where a name cost up to 35%.At the end, because offsets are only known once the items are written — a leading index would mean buffering the whole sequence before emitting a byte, fatal for a log. ZIP's central directory and Parquet's footer sit at the end for the same reason.
Found through its own last element, not a magic trailer. An 8-byte byte string always encodes as
0x48plus 8, so a sealed file ends with 9 predictable bytes however large the index is. Those bytes are ordinary CBOR — the file stays a valid sequence thatcbor2andciboriumread in full, getting every data item plus one tagged value they can ignore. There is a test for exactly that.One mechanism, two levels
The sequence offsets are simply the node at the sentinel offset −1 — it has no container header on the wire but behaves like one, so there is a single uniform node list rather than two shapes. Everything else is a node per container: an array indexes positionally in O(1), a map with sorted keys binary-searches in O(log n) comparing encoded key bytes, decoding no key and touching no value.
Tags are descended through, not stepped over. A set is tag 258 around an array and a record tag 27 around a map; skipping tags left exactly those uncovered, and descending costs nothing since the walk covers the same bytes anyway.
Slots are deltas, in the narrowest type that holds them
Offsets inside a container ascend, so consecutive differences are small and nearly uniform while the absolutes are large and unbounded. Each slot goes out as differences from the previous entry — from the container's own offset for the first — in the narrowest of a byte string, sint16 (tag 77) or sint32 (tag 78).
No new format surface. All three already round-trip through the codec, and the CBOR element type is the width declaration, so there is no per-entry flag of the kind PostgreSQL needs for its
JEntryarray. Postgres reads offsets in place out of a TOAST'd datum and pays a prefix sum per probe; boring materialises the index once when it loads and expands there, solookup-map,nthandnode-slotstill read a plainint[]and nothing on the hot path changed.Absolute int32 offsets cost 781 KB at stride 1, 10.9% of that file; deltas cost 195 KB. That 4× is the point — it makes stride 1, an index with no scan component at all, a defensible choice rather than a curiosity. At stride 16 the index was already 0.38% of the file and this changes nothing anyone would notice.
Narrowing also breaks proportionality: doubling the stride halves the anchor count but can double the width, so size falls in steps and there are bands where a denser index is nearly free.
A reader must therefore treat an int32 slot as deltas too — absolutes and deltas are indistinguishable on the wire. That is why this landed here rather than as a follow-up: merging to main publishes the format, since
versionis0.1.<git-count-revs>andtools/deployruns on every push to main.The writer captures the index; it does not walk for it
write-seq!builds the index while encoding. The writer already knowspos— the offset it is about to write to — and knows a container's entry count before emitting a byte of it, so the nodes fall out of encoding. Nothing in the index needs a subtree's length, only where its entries start, so there is nothing to back-patch. In a length-prefixed format this would need a second pass; CBOR's element counts make the write side free and the read side expensive, and this takes the good half of that trade.50 000 records through a
BufferedOutputStream, against the same write with no index:Stride 1's remainder is not capture — it is delta-encoding and emitting a 50 000-entry slot at the end, which the walk paid too.
build-indexstays. Indexing a value that is already encoded — re-indexing after a compaction, indexing a file somebody else wrote — has no other path, and it is now also the oracle the captured index is tested against.The two differ, deliberately. The walk indexes containers on the wire, and boring puts several there that no user wrote: a sorted-map is tag 27 around a two-element
[name, map], a shaped array is[keys, rows]. On the wire those are indistinguishable from a user's own two-element vector, so the walk indexes them and the writer skips them. That is a subset, not a disagreement — a node for[name, map]is pure overhead, and the index is an optimisation. Pinned contract: every captured node is byte-identical to one the walk found (always), and the two agree completely once:index-minexcludes frames. Every frame boring emits has 3 entries or fewer, so 4 is the boundary and the default of 16 clears it.boring.writer-index-testis generative and found that divergence within a minute. It shrank to(sorted-map)— which prints as{}, so every attempt to reproduce it by hand passed.Twenty correctness defects, found by review and fixed here
Three rounds of independent review, two reviewers each. The invariant below was false in six ways to begin with, four of which fired on ordinary data at the shipped defaults with no corruption involved. Fourteen more followed, including five caused by the fixes themselves — the rounds kept earning their cost.
Every defect has a regression test in
boring.index-robustness-test.On which point I have to correct an earlier version of this description, which claimed every test was verified to fail without its fix. That verification reverted all of
src/at once, which proves the batch is covered and not each fix. A per-fix sweep — revert one change, rebuild, require its own test to fail — found two tests that passed with their fix removed, in a commit whose message asserted otherwise. Both are repaired, and the per-fix sweep is now the method.sorteddecided from an anchor samplenildespite existingnthwith no sequence node:canonical-rfc7049marked sortedArrayIndexOutOfBoundsExceptionat the callerSeven more from the same reviews, outside the index — two of them on
mainand therefore in a released version:boring.navdropped every decode option (:registryignored,:max-depthunenforced); nested tags escaped:max-depthinskipwhilereadcharged for them, so the cheap path routed around a security bound; a failed read poisoned the shared Reader's depth; canonical maps could emit duplicate keys;encodeFallbackwas not inherited by the canonical scratch writer; collections that lie aboutsize()were written anyway; andwrite-seq!'s 3-arity ignored the writer's options.Also hardened: an index offset past 2 GiB now refuses rather than wrapping negative, a failed indexed write no longer leaves capture running, and
idxResetreleases the anchor arrays it used to pin.The testing lesson, since it explains how these survived: the differential test compares the two index builders against each other, so the worst defect — present in both — was invisible to it. An oracle must be independent of what it checks; for
sortedthat means decoding the map, not cross-checking two implementations that share an author and an assumption.The index is not load-bearing for correctness — with one stated limit
That is what the tests pin. A corrupted pointer, a truncated file, an appended-to file, a file that merely ends in the right 9-byte shape, and a frame that decodes but whose parts disagree all still produce right answers. Truncation surfaces as a typed error, not a fabricated answer.
The limit, stated rather than implied: this covers a missing, stale, truncated or randomly corrupt index. It does not cover a crafted one. The reader checks that a node's parts agree — the container is really at that offset with the count it claims, its first anchor is that container's first entry, anchors ascend and stay in range — but verifying that every anchor is a real entry boundary is O(n) per container, and verifying
sortedmeans reading every key. Both are precisely the work the index exists to avoid. So the index frame is a trust boundary: integrity of the index is integrity of the document, which is the exposure the data section already has since CBOR carries no checksum.doc/SHAPES.mdhas the reasoning.Detection is not only about speed: the index is itself a top-level item, so without recognising it
nav/itemswould yield it as data. Which is exactly what an empty sealed sequence did until the last commit — its data section is 0 bytes, so its back-pointer is 0, and the reader demanded a positive one.Two knobs, and which one matters
:index-mindominates, and its first default of 2 was badly wrong. Real data is mostly small containers, each costing an offset, a count, a slot and a flag. On 2 000 two-field records, indexing everything cost 76% of the file; indexing only containers of 8+ cost 1.3% — and was faster, the smaller index also fitting in cache. Default is now 16.Where it still loses, in the benchmark output rather than buried: finding a node is itself a binary search over the container list, once per level. For containers that are narrow and whose entries are cheap to skip, walking beats jumping — 64-key levels measured 2× slower indexed. Raising
:index-minabove such widths is the fix.Also here
mmap/mmap-items, which was missing:mmap-sourcereturns a cursor, the single-value shape, with no equivalent for a sequence — the headline case for an index over a mapping. The mmap test pins delta expansion through the segment accessor rather than thebyte[]fast path, where a disagreement would not throw but would silently seek to the wrong offset.writebenchmark (-m nav write) that measures against a realBufferedOutputStreamand warms every path before timing any of them. Both rules exist because breaking them produced wrong answers here: aproxysink pads the denominator, and two warmup iterations put whichever case ran first at a disadvantage large enough to report "no index" as slower than a stride-1 index.Map,SetandCollectioncases in the general dispatch now delegate rather than carrying inlined copies of the loops inwriteMapValueandwriteSeqAsArray. The first version of the index hook instrumentedwriteMapValueand indexed nothing, because plain Clojure maps never reached it.Before merging
tools/deployruns on the orb's executor, andclj -T:build deployrefuses to publish a jar withoutSegmentSource, which a JDK older than 22 cannot build. If the orb's image is JDK 21 the release fails that gate — correctly, but it needs the orb's JDK confirmed or the call pinned to a 22+ image first. Not verifiable from the machine this was written on.