nsqd: better diskqueue error backpressure #437
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| select { | ||
| case c.memoryMsgChan <- m: | ||
| default: | ||
| // TODO: re-use this buffer |
There was a problem hiding this comment.
I'd like to figure out how to re-use this buffer before we merge...
this change accomplishes a few things...
1. goroutines calling topic.Put*() via HTTP or TCP
will now be able to directly detect failed diskqueue writes,
giving the type of feedback you need to retry, without
any asynchronous interaction.
2. related to (1), goroutines (and thus clients) will experience
*direct* backpressure when messages are written to disk.
3. removes an extra goroutine per topic and an extra channel
send/receive per PUB. the aggregate effect on benchmarks
is significant because this setup has far better concurrency
characteristics (41% faster):
$ GOMAXPROCS=4 ./bench.sh
PUB: 2014/08/15 10:17:37 duration: 996.190589ms - 191.464mb/s - 1003823.978ops/s - 0.996us/op
|
RFR @jehiah |
jehiah
added a commit
that referenced
this pull request
Aug 18, 2014
nsqd: better diskqueue error backpressure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a continuation of the work done in #422, returning the
errordirectly to thePUBin question, thus providing a better backpressure mechanism (vs. asynchronous as of #422).Simultaneously, it happens to significantly improve performance (41% on
PUBbenchmark) by removing the topic'srouter()goroutine (see first commit msg for details, but basicallyrouter()was effectively serializingPUBrequests):