[NIDX-01B] Count committed generations with deletions

[hanahmily]

[NIDX-01B] Count committed generations with deletions

Parent workstream: #14002
Blocked by: #14008

Summary

Extend the live native inverted.ReadOnlyDocCount cutover from NIDX-01A to multiple segments, snapshot deletion masks, newest-valid generation selection, and concurrent legacy publication. This slice adds no document visitation or query surface.

Blocked by

NIDX-01A (#14008) must be merged before this work starts. An open pull request is insufficient.

Pre-agreed test seam

Test only:

func inverted.ReadOnlyDocCount(path string) (int64, error)

Generation enumeration, snapshot records, roaring deletion masks, and segment accounting remain private.

Independent fixture

Extend the legacy corpus with:

• valid generation 10, referencing two segments containing five physical documents;
• one snapshot deletion, for doc-22;
• invalid newer generation 11, referencing a missing segment; and
• arbitrary reserved CRC32 bytes.

The declared selected generation is 10; the visible count is the literal 4.

Requirements

R1. ReadOnlyDocCount selects the newest structurally complete generation and returns 4. It never combines records from generations 10 and 11.

R2. Count is computed from segment counts and deletion masks without decoding stored fields or materializing document results.

R3. While a retained legacy writer publishes a complete generation adding doc-26, concurrent count calls return only 4 or 5. They never observe a partial generation, create an index-local runtime file, or fail because the writer is open.

R4. Missing segments, invalid deletion ordinals, truncated roaring data, and arithmetic overflow produce bounded typed errors. Reserved CRC32 values remain ignored.

RED and end-to-end contract

• Boundary RED: the NIDX-01A implementation is exercised against generation 10/11; it must fail until multi-segment deletion and generation selection are implemented.
• E2E RED: keep the legacy writer open, publish doc-26 while repeated production counts run, and assert the observed set is a subset of {4,5} and eventually contains 5; directory readers never expose an intermediate count.

Acceptance criteria

• The extended fixture and provenance manifest declare generation IDs, segment membership, the deleted ordinal, file hashes, and expected count 4 independently of native code.
• go test ./pkg/index/inverted/... -run 'Test(ReadOnlyDocCountCommittedGenerations|E2EReadOnlyCountDuringPublication)' -count=1 passes, including under -race.
• Before/after inventories prove the reader changes no bytes, mtimes, directory entries, or runtime files.
• make test-ci PKG=./pkg/index/inverted/..., make build, and make pre-push pass.

Scope

Packages: pkg/index/inverted and its private native snapshot/deletion reader.

Out of scope: stored fields, dictionaries, term postings, document walk, doc values, sort/search-after, writers, and merge.

Compatibility and rollback

This slice emits no index bytes. Roll back ReadOnlyDocCount to its retained legacy reader without a writer drain or directory conversion.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01