Background
SkipCache (src/skip_cache.rs) stores precomputed sibling-span offsets so that repeated lookups on the same container avoid re-walking the full brace tree. Currently the cache grows unboundedly for the lifetime of the document.
For most workloads (parse-once, read a few fields, discard) this is fine. For long-lived documents with very large or deeply-nested JSON (hundreds of distinct containers accessed), the cache can accumulate significant memory.
Proposal
Add an LRU eviction policy to SkipCache so that the cache is bounded to a configurable maximum number of entries (e.g. 256 or 512 spans). The lru crate or a simple hand-rolled clock eviction is sufficient.
Why it was deferred
No concrete memory-pressure report from production workloads. The concern is theoretical: mmap'd pages stay lazily faulted, so over-reservation doesn't cause real RSS growth until pages are touched.
When to revisit
When a production user reports unexpected memory growth on documents with many distinct containers, or when profiling shows SkipCache as a significant contributor to peak RSS.
Notes
- Eviction adds a lookup cost; benchmark before/after on the existing multimodal bench to confirm no throughput regression on the common "read a few fields" case.
- The cache is behind a
RefCell (single-threaded); no locking needed.
Background
SkipCache(src/skip_cache.rs) stores precomputed sibling-span offsets so that repeated lookups on the same container avoid re-walking the full brace tree. Currently the cache grows unboundedly for the lifetime of the document.For most workloads (parse-once, read a few fields, discard) this is fine. For long-lived documents with very large or deeply-nested JSON (hundreds of distinct containers accessed), the cache can accumulate significant memory.
Proposal
Add an LRU eviction policy to
SkipCacheso that the cache is bounded to a configurable maximum number of entries (e.g. 256 or 512 spans). Thelrucrate or a simple hand-rolled clock eviction is sufficient.Why it was deferred
No concrete memory-pressure report from production workloads. The concern is theoretical: mmap'd pages stay lazily faulted, so over-reservation doesn't cause real RSS growth until pages are touched.
When to revisit
When a production user reports unexpected memory growth on documents with many distinct containers, or when profiling shows
SkipCacheas a significant contributor to peak RSS.Notes
RefCell(single-threaded); no locking needed.