Remove dictionary rows when no document is referring to it#1314
Remove dictionary rows when no document is referring to it#1314tanzhang0504 wants to merge 3 commits intoblevesearch:masterfrom
Conversation
…nger referred by any document.
| doc := document.NewDocument("1") | ||
| doc.AddField(document.NewTextField("name", []uint64{}, []byte("test"))) | ||
| err = idx.Update(doc) | ||
| doc.AddField(document.NewTextField("name1", []uint64{}, []byte("test1"))) |
There was a problem hiding this comment.
I enhanced the test to include one index that's shared by two documents, and the other is not. I'm also happy to revert to the original test setup if you think this hurts readability.
|
Similar issue was discussed in blevesearch/blevex#30 and blevesearch/blevex#44 |
Hi Aman, thanks for the pointers. I'm also wondering is it possible to not generate dictionary rows altogether? Seems like it's needed by bleve to look up the terms. I did an experiment to not insert dictionary rows (by removing this line - bleve/index/upsidedown/upsidedown.go Line 287 in 5b9d775 |
|
@mschoch Just wondering any comment on my PR? I saw the past discussions in blevesearch/blevex#30 as pointed out by Aman, and seems like the suggested solution is to add a Compact() function for each DB store and rely on callers to invoke that function. Just wondering will that add too much burden on the caller instead of fixing the leak at the store level? Any feedback is greatly appreciated! |
2 participants
Hi - we have a project that uses bleve in-memory index based on gtreap implementation. We noticed a similar behavior that dictionary rows keep grow even after all the documents referring to them are deleted. I found #374, and seems that someone made a temporary only to leveldb, which does not apply to bleve in-memory index use-case.
So I've made this PR to systematically fix #374 for all different index store implementations. Looking forward to your feedback!