fix(#791): eliminate spTxIndex, drop singleton subpaths to slash startup memory#796
Closed
Kpa-clawbot wants to merge 3 commits intomasterfrom
Closed
fix(#791): eliminate spTxIndex, drop singleton subpaths to slash startup memory#796Kpa-clawbot wants to merge 3 commits intomasterfrom
Kpa-clawbot wants to merge 3 commits intomasterfrom
Conversation
added 3 commits
April 19, 2026 00:03
Remove the spTxIndex map (map[string][]*StoreTx) that stored per-subpath transmission pointer slices. This was the largest single heap consumer during startup on databases with many unique subpaths. Replace the only read site (GetSubpathDetail) with an O(packets) scan that matches subpaths on the fly. This is acceptable because the endpoint is only called on drill-down, not listing. Remove all write sites (addTxToSubpathIndexFull, removeTxFromSubpathIndexFull) and collapse them back into the simpler addTxToSubpathIndex/removeTxFromSubpathIndex functions that only maintain counts.
After the initial subpath index build, iterate and delete all entries where count==1. At scale, 60-70% of subpath keys are singletons that appear only once — they add no analytical value and consume significant memory (each entry is a string key + map bucket overhead). Newly-seen singletons during incremental ingest will remain in the index until the next restart. This is acceptable since singleton subpaths are noise for ranking/display purposes. Add a startup log line showing how many entries were kept vs dropped.
Remove perSubpathEntryBytes constant and all O(path²) subpath cost calculations from estimateStoreTxBytes and estimateStoreTxBytesTypical. Update comments to note that spTxIndex was eliminated in #791 and singleton subpath entries are dropped from spIndex.
Closed
32 tasks
Owner
Author
|
Closing in favor of #799, which targets the actual heap dominator. Profiling staging revealed the OOM in #791 is driven by JSON-decoded See #799 for the proposed refactor (membership index + on-demand API decode), test plan, and acceptance criteria. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminate the two largest heap consumers in the subpath index to prevent OOM during startup on databases with many unique subpaths.
Changes
Eliminate
spTxIndexentirely — Themap[string][]*StoreTxthat stored per-subpath transmission pointer slices has been removed. The only read site (GetSubpathDetail) now scans packets on the fly, which is acceptable since it's only called on drill-down, not listing.Drop singleton subpath entries — After the initial index build, entries with count==1 are pruned. At scale, 60–70% of subpath keys are singletons that add no analytical value. Newly-seen singletons during incremental ingest remain until the next restart.
Update memory-accounting — Remove
perSubpathEntryBytesconstant and O(path²) subpath cost calculations fromestimateStoreTxBytes/estimateStoreTxBytesTypical. Update comments to reflect the changes.Add startup log line —
[store] Subpath index: kept N/M entries (dropped K singletons)Estimated Memory Savings
Tests
GetSubpathDetail) returns correct match counts (existing + updated test)TestSubpathSingletonDrop)TestSubpathEmptyDB)This replaces the approach in draft PR #716.
Closes #791