A concurrent version of trie index #334
Conversation
|
Is this PR still relevant, @bom-d-van @azhiltsov ? |
|
We are currently running two types or indexes in production
Since this feature is promising to cut the memory footprint and also remove the need in metric scan every 5 minutes or so, I see a lot value in it. However I do not know how much of the time and effort from the side of @bom-d-van will be required to finish it. |
|
@deniszh @azhiltsov Thanks for the input. If there is a chance of adoption. I am happy to pull it off. Last time when I was testing it, there seemed to be some memory leaks going on. Again, gonna take some time to have updates. I think I was attacking too many issues/features in the go graphite stack at the same time. Sorry if it bothers you. (I was constantly looking for interesting stuff to work on but new ideas keep showing up). And big thanks for @deniszh for doing all the following up and tough works. |
|
Again, much appreciated to @bom-d-van for pushing this forward, no rush, let's return to this later. Thanks! |
bom-d-van
force-pushed
the
ctrie
branch
2 times, most recently
from
5414564
to
3de57e9
Compare
* file nodes are no longer a global variable as pruning required a mark value (extra memory costs: o(n)) * pruning childrens on live trie nodes rather than the walking copy (this was causing memory leak as many old nodes are properly pruned)
* rename trieNode.mark to gen * prune: fix merging logics * insert: fix incorrect gen syncing for splits * imporve prune and real test (check both dump and node count after prune)
* fix ci errors * handle potential index out of bound in all the trie walking funcs * refactor test logs
|
There are three new features introduced in this PR, two for trie index: edit: comment repost: #374 (comment) |
|
Thanks, merging this, @bom-d-van ! |
This is a follow up to #332
With this change, go-carbon no longer requires two versions of trie tree in memory. So memory overhead could be cut down even further in theory. More measurement needed to have some numbers.
This could also make new metrics available in "realtime" without waiting for re-scanning-and-indexing to be finished.