Remove incomingMessage from the incomingQueue only once #321
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What are we trying to achieve with this PR:
Fixing
NoSuchElementExceptionwhen using theDirectExecutorServiceascompressionExecutor.Initial exception
Issue analysis
Currently, we are using
SyncWriterin order to write high amount of data into the YDB topicWe've noticed that during peak load per thread (when the availableSizeBytes starts getting exceeded), we get
NoSuchElementExceptionwhich cause the write session to reboot and our calls to time outSeems like it is happening due to the chain of calls
onWriteResponse -> free -> acceptMessageIntoSendingQueue -> moveEncodedMessagesToSendingQueue -> free
All elements in this chain are executed in the same thread (with the acquired lock on incomingQueue) due to the usage of the Guava's DirectExecutorService
Proposal
Pop the message from the
incomingQueueonly once during the completion stage of theincomingMessage.future(asincomingQueue.remove()can only target this exactincomingMessage)