Improve stream reader metrics cleanup #3340
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gerhard
force-pushed
the
improve-stream-reader-metrics-cleanup
branch
from
b288ffc
to
97704ec
Compare
Otherwise metrics will not get cleaned up correctly when processes crash. It's also tidier to do this in a single place, in terminate/3 Pair: @kjnilsson Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Pair: @kjnilsson Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Pair: @kjnilsson Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
This ensures that only nodes that are ready to host stream members are included in the election. This avoids continuous restart attempts when the rabbit application is stopped.
Some logs used ~p to format a full stack trace. Given these warnings are emitted during any nodedown this unnecessarily pollutes the logs. Trimmed using ~W instead.
Also increased the tick timeout to avoid checking for new rabbit nodes to auto add too often. Also increased sleep times for nodedowns to retry less often.
When the server initiate connection close.
gerhard
force-pushed
the
improve-stream-reader-metrics-cleanup
branch
from
97704ec
to
c240ec2
Compare
Rather than sleeping for 6 seconds, we want to check that replica recovered multiple times within 30 seconds, and either eventually succeed, or fail if this does not recover within 30 seconds, the default await_condition time interval. Pair: @kjnilsson Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
|
@Mergifyio backport v3.9.x |
|
Command
|
michaelklishin
added a commit
that referenced
this pull request
Sep 1, 2021
Improve stream reader metrics cleanup (backport #3340)
gerhard
added a commit
that referenced
this pull request
Sep 1, 2021
I've missed adding notes to the release part of #3340 I am going to add a new item to the PULL_REQUEST_TEMPLATE.md checklist so that we get reminded about it next time. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When stream readers crash, we want to cleanup metrics, otherwise stream consumer and publisher metrics will diverge from reality, and nodes will report a higher number of these than there are in reality.
We cleaned up some logs in the process, and the
replica_recoverytest in rabbit_stream_queue SUITE was made more reliable.