fix(dataloader): #315 SVAR2 variant-windows double_buffered slot under-allocation (Layer 1)

[d-laub]

Summary

Completes the deferred Layer 1 of #315: Dataset.to_dataloader(mode="double_buffered") over flat variant-windows failed deterministically at scale on SVAR2-format datasets with RuntimeError: ProducerError (ValueError): buffer is smaller than requested size (later SlotOverflowError). The shared-memory slot was sized from a per-instance byte estimate that under-counted the real serialized payload on the SVAR2 reconstruction path.

Root cause

The reported corpus is SVAR2-format-2.0.0 and reconstructs through Svar2Haps (a Haps subclass), not Haps. Dataset._output_bytes_per_instance's variant-windows branch derives its estimate from three terms — n_vars_total, ref_span, and alt_alleles — all read from Svar2Haps.genotypes / Svar2Haps.variants. For Svar2Haps those are permanently-empty SVAR1-shaped placeholders, constructed once at open time only so scattered isinstance(_, Haps) checks keep working; they are never populated, because SVAR2 reconstructs read-bound from the on-disk .svar2 store via Rust FFI (decode_variants_from_svar2_readbound), not through an in-memory sparse genotypes table.

Effect: all three terms collapse to 0 for every instance, so the estimate degenerates to a flat, content-independent 32 bytes/instance while the real payload scales with variant content (up to ~17 KB/instance on the real corpus). The 40×7089 chunk overran its estimate-sized slot → overflow. This is not a formula bug and not a p_eff/ploidy bug — it is a wrong data source.

Fix

• Added Svar2Haps.measure_variant_payload(...): a shared per-instance measure that runs the same decode_variants_from_svar2_readbound + unphased_union → p_eff fold (var_off[::P]) the reconstructors already use, returning (n_vars_total, ref_span_sum, alt_bytes_sum).
• Dataset._output_bytes_per_instance now sources those three terms from that shared entry point when isinstance(haps_obj, Svar2Haps), so the estimator and reconstructor derive their variant accounting from the same code and cannot drift apart again. Applied to both the variant-windows branch (the reported repro) and the sibling variants (non-window) branch, which had the identical defect.
• The Haps (SVAR1/VCF/PGEN) estimate is byte-for-byte unchanged — the fix is gated on Svar2Haps.

Test coverage

tests/unit/test_slot_fit_property.py gains test_slot_fit_svar2_backend: a SVAR2-backed dataset opened via the released Dataset.open + to_dataloader/reconstruct path (the coverage gap that let #315 through), asserting est + slot_overhead ≥ real serialized payload across ref/alt × unphased_union for both variant-windows and variants output modes — red before the fix, green after. Existing Haps-path cases (dummy/VCF/PGEN/SVAR1) pass unchanged.

Real-corpus confirmation

Re-ran the exact reported repro (40 regions × 7089 samples, flat variant-windows, ref="window", alt="allele", unphased_union=True, jitter=0, flank_length=128, default 2 GiB buffer_bytes, batch_size=4096) against the real Hartwig SVAR2 corpus post-fix: to_dataloader(mode="double_buffered") now iterates batches with no SlotOverflowError/ProducerError.

Notes

• Scope: the released double_buffered path over a written, file-backed dataset — normal main-targeted work, not StreamingDataset.
• Out of scope (unchanged): buffered/manual modes, Bug A (realign_tracks), the producer-side grow-or-split escalation.
• jitter>0 on SVAR2 variant-windows/variants remains guarded by the existing NotImplementedError ("right-clip") in _svar2_haps.py, so the measure intentionally does not widen by jitter (that path is unreachable).
• Design/pin docs: docs/superpowers/specs/2026-07-23-svar2-variant-windows-slot-fit-design.md and …-phase0-realcorpus-findings.md.

🤖 Generated with Claude Code