New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAFKA-4379 Followup: Avoid sending to changelog while restoring InMemoryLRUCache #2908
Conversation
guozhangwang
commented
Apr 25, 2017
•
edited
edited
- Added a flag to indicate if it is restoring or not in the LRU Store; since we only have a restore callback we have to set it each time applying the change.
- Fixed the corresponding unit test, plus some minor cleaning up.
ping @norwood @dguy for reviews. The reason that I think it is safe to remove the listener is that with either log truncation and log compaction, we can actually prove that after replaying the lru store's image will not contain any unexpected records, i.e.: any record that will show up in the restored lru will definitely be in the original lru, even with With that, I think we then do not need to log the evicted records anymore from the changelog topic, and it is safe to just remove this piece of logic here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@guozhangwang what is the downside of this? Is it that in the case it is backed by a compacted topic that items won't ever be removed as tombstones aren't sent?
time for a new compaction policy based on message count |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
@norwood @dguy @mjsax I thought about that too. What I concluded was that if we record the evicted record along with log compaction, we actually will not get correct restored state compared with the original lru store ("state" include the ordering of the filled records which will affect future eviction decisions after restoration). Also I did some simple test to verify that with recorded eviction record, restored state may actually be incorrect, so admittedly the downside of ever growing changelog is worse, but that is necessary for correctness. On the other hand, since as we have discussed, we can write the offset in checkpoint file as well for lru because of the proven property mentioned above to reduce restoration latency (this is a niche optimization for lru only so not to much work to add). |
Chatted with @norwood offline, and we decided to go with the alternative direction to still log evicted records only after restoration. |
Refer to this link for build results (access rights to CI server needed): |
6343783
to
3f0d523
Compare
if (listener != null) listener.apply(key, eldest.getValue()); | ||
return true; | ||
boolean evict = super.size() > maxCacheSize; | ||
if (evict && !restoring && listener != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do the updates on restoring
from the StateRestoreCallback
always happen in the same thread?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I'll merge after @guozhangwang changes the title.
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
local unit test passed. |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |