-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C++] Fix request timeout for GetLastMessageId doesn't work #12586
Merged
merlimat
merged 5 commits into
apache:master
from
BewareMyPower:bewaremypower/cpp-correct-timeout
Nov 3, 2021
Merged
[C++] Fix request timeout for GetLastMessageId doesn't work #12586
merlimat
merged 5 commits into
apache:master
from
BewareMyPower:bewaremypower/cpp-correct-timeout
Nov 3, 2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I'll fix the compilation error soon. |
9bf793b
to
3d2a4fd
Compare
merlimat
reviewed
Nov 2, 2021
Mark it as WIP first. I'll fix the tests in local env before pushing the commits. And I will revert the refactoring of Future. |
merlimat
reviewed
Nov 3, 2021
merlimat
approved these changes
Nov 3, 2021
hangc0276
pushed a commit
that referenced
this pull request
Nov 4, 2021
* Fix request timeout for GetLastMessageId doesn't work * Fix CentOS 7 build error * Revert refactors * Remove redundant clear for listeners * Use swap instead of move (cherry picked from commit a54c6c0)
eolivelli
pushed a commit
to eolivelli/pulsar
that referenced
this pull request
Nov 29, 2021
…2586) * Fix request timeout for GetLastMessageId doesn't work * Fix CentOS 7 build error * Revert refactors * Remove redundant clear for listeners * Use swap instead of move
codelipenghui
pushed a commit
that referenced
this pull request
Dec 20, 2021
* Fix request timeout for GetLastMessageId doesn't work * Fix CentOS 7 build error * Revert refactors * Remove redundant clear for listeners * Use swap instead of move (cherry picked from commit a54c6c0)
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
cherry-picked/branch-2.8
Archived: 2.8 is end of life
cherry-picked/branch-2.9
Archived: 2.9 is end of life
doc-not-needed
Your PR changes do not impact docs
release/2.8.2
release/2.9.2
type/bug
The PR fixed a bug or issue reported a bug
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
#11600 adds the timeout for GetLastMessageId request by using
sendRequestWithId
instead ofsendCommand
. However, it's still incorrect. Because when the request timeout exceeds, the future that is completed withResultTimeout
is whatsendRequestWithId
returns but not thepromise.getFuture()
. Therefore, even if the request was not responded inoperationTimeout
seconds, the future returned bynewGetLastMessageId
would still be not completed.Besides, when I tried to complete the
promise
insendRequestWithId
's callback, I found a deadlock issue ifServerCnx#handleGetLastMessageId
hang forever (I just add a longsleep
call in this method).We can see
Promise::setFailed
stuck inClientConnection::close
:It's because the future's callback is called in
setFailed
. However, the callback also callssetFailed
, which tries to acquire the same lock that is not reentrant. So deadlock happens.Modifications
Refactor the
Future
/Promise
infrastructure. The current design is too old and the code style is bad. The important things of the refactoring are:completed_
field (the originalcomplete
field) to an atomic variable. So when checking if the future is completed, no lock is required.listeners_
out of the locked code block. So that each listener (callback) will be triggered without acquiring any lock.conditional_variable::notify_all()
out of the locked code block as well. The notifying thread does not need to hold the lock, see https://en.cppreference.com/w/cpp/thread/condition_variable/notify_all.InternalState
so thatFuture
andPromise
only need to call them directly.Then add a
PromiseTest
to protect the refactoring. Based on the refactor, just add a callback tosendRequestWithId
innewGetLastMessageId
to make sure the request timeout works.Verifying this change
It's hard to simulate the operation timeout. So I have to add the following code to
ServerCnx#handleGetLastMessageId
and run a reader to call
hasMessageAvailable
with 3 seconds operation timeout. After it failed, the client exited after 6 seconds, which is twice the operation timeout. The logs are: