Fix for when consumer would stop working due to errPartialCache returned from fileStore. #2761
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…urn hint about clearing cache. Signed-off-by: Derek Collison <derek@nats.io>
… certain situations. Related to #2732 The filestore would release a msgBlock lock while trying to load a cache block if it thought it needed to flush pending data. With async false, this should be very rare but was possible after careful inspection. I constructed an artificial test with sleeps throughout the filestore code to reproduce. It involved having 2 Go routines that were through and waiting on the last msg block, and another one that was writing. After the write, but before we flushed after releasing the lock we would also artificially sleep. This would lead to the second read seeing the cache load was already in progress and return no error. If the load was for a sequence before the current write sequence, and async was false, the cache fseq would be higher than what was requested. This would cause the errPartialCache to be returned. Once returned to the consumer level in loopAndGather, it would exit that Go routine and the consumer would cease to function. This change removed the unlock of a msgBlock to perform and flush, ensuring that two cacheLoads would not yield the errPartialCache. I also updated the consumer in the case this does happen in the future to not exit the loopAndGather Go routine. Signed-off-by: Derek Collison <derek@nats.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stabilize filestore to eliminate sporadic errPartialCache errors under certain situations.
The filestore would release a msgBlock lock while trying to load a cache block if it thought it needed to flush pending data. With async false, this should be very rare but was possible after careful inspection and could lead under the right conditions to a consumer getting an errPartialCache returned trying to load a message.
Resolves #2732 (Hopefully)
/cc @nats-io/core