[broker] Increase timeout for loading topics #6750
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
addisonj
force-pushed
the
load-increase-timeout
branch
2 times, most recently
from
b394070
to
ed1a9ec
Compare
| @@ -861,7 +861,8 @@ public PulsarAdmin getClusterPulsarAdmin(String cluster) { | |||
| protected CompletableFuture<Optional<Topic>> loadOrCreatePersistentTopic(final String topic, | |||
| boolean createIfMissing) throws RuntimeException { | |||
| checkTopicNsOwnership(topic); | |||
| final CompletableFuture<Optional<Topic>> topicFuture = futureWithDeadline(); | |||
| // this timeout needs to be extra long in the case of topics with many replication clusters | |||
| final CompletableFuture<Optional<Topic>> topicFuture = futureWithDeadline(5L, TimeUnit.MINUTES, new TimeoutException("Failed to load topic within timeout")); | |||
There was a problem hiding this comment.
It's better to make the timeout of the topic loading configurable in the broker.conf
There was a problem hiding this comment.
+1 we should make this setting configurable.
In apache#6489, a timeout was introduced to make sure calls into the BrokerService finish or error out. However, this timeout is too low by default when loading topics that have many replicated clusters. Loading replicated topics is quite an expensive operation, involve global ZK lookups and the start of many sub-processes. While we would hope it finishes in 60 seconds we want to safe. Long term, it may make sense to break out this operation into more steps where each step can have it's own timeout
addisonj
force-pushed
the
load-increase-timeout
branch
from
ed1a9ec
to
209e6d0
Compare
|
@sijie @codelipenghui added the option, let me know if there is anything else! |
|
/pulsarbot run-failure-checks |
|
@addisonj Could you please help add the configuration to the broker.conf? So that users can find it easier. The change looks good to me. |
codelipenghui
approved these changes
May 7, 2020
sijie
approved these changes
May 7, 2020
jiazhai
pushed a commit
that referenced
this pull request
May 8, 2020
In #6489, a timeout was introduced to make sure calls into the BrokerService finish or error out. However, this timeout is too low by default when loading topics that have many replicated clusters. Loading replicated topics is quite an expensive operation, involve global ZK lookups and the start of many sub-processes. While we would hope it finishes in 60 seconds we want to safe. Long term, it may make sense to break out this operation into more steps where each step can have it's own timeout Co-authored-by: Addison Higham <ahigham@instructure.com>(cherry picked from commit 6854b00)
Huanli-Meng
pushed a commit
to Huanli-Meng/pulsar
that referenced
this pull request
May 27, 2020
In apache#6489, a timeout was introduced to make sure calls into the BrokerService finish or error out. However, this timeout is too low by default when loading topics that have many replicated clusters. Loading replicated topics is quite an expensive operation, involve global ZK lookups and the start of many sub-processes. While we would hope it finishes in 60 seconds we want to safe. Long term, it may make sense to break out this operation into more steps where each step can have it's own timeout Co-authored-by: Addison Higham <ahigham@instructure.com>
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
In apache#6489, a timeout was introduced to make sure calls into the BrokerService finish or error out. However, this timeout is too low by default when loading topics that have many replicated clusters. Loading replicated topics is quite an expensive operation, involve global ZK lookups and the start of many sub-processes. While we would hope it finishes in 60 seconds we want to safe. Long term, it may make sense to break out this operation into more steps where each step can have it's own timeout Co-authored-by: Addison Higham <ahigham@instructure.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #6489, a timeout was introduced to make sure calls into the
BrokerService finish or error out. However, this timeout is too low by
default when loading topics that have many replicated clusters.
Loading replicated topics is quite an expensive operation, involve
global ZK lookups and the start of many sub-processes. While we would
hope it finishes in 60 seconds we want to safe.
Long term, it may make sense to break out this operation into more
steps where each step can have it's own timeout