[client] Limit the number of times lookup requests are redirected #7096
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
massakam
added
type/enhancement
codelipenghui
approved these changes
May 29, 2020
sijie
approved these changes
May 29, 2020
Huanli-Meng
pushed a commit
to Huanli-Meng/pulsar
that referenced
this pull request
Jun 1, 2020
Master Issue: apache#7041 ### Motivation When a leader broker is restarted, some producers for topics owned by that broker may not be reopened on the new broker. When this happens, message publishing will continue to fail until the client application is restarted. As a result of the investigation, I found that lookup requests sent by the producers in question are redirected more than 10,000 times between multiple brokers. When a lookup request is redirected, `BinaryProtoLookupService#findBroker()` is called recursively. Therefore, tens of thousands of redirects will cause `StackOverflowError` and `BinaryProtoLookupService#findBroker()` will never complete. ### Modifications Limit the number of times a lookup is redirected to 100. This maximum is user configurable. If the number of redirects exceeds 100, the lookup will fail. But `ConnectionHandler` retries lookup so that the producer can eventually reconnect to the new broker.
Huanli-Meng
pushed a commit
to Huanli-Meng/pulsar
that referenced
this pull request
Jun 1, 2020
Master Issue: apache#7041 ### Motivation When a leader broker is restarted, some producers for topics owned by that broker may not be reopened on the new broker. When this happens, message publishing will continue to fail until the client application is restarted. As a result of the investigation, I found that lookup requests sent by the producers in question are redirected more than 10,000 times between multiple brokers. When a lookup request is redirected, `BinaryProtoLookupService#findBroker()` is called recursively. Therefore, tens of thousands of redirects will cause `StackOverflowError` and `BinaryProtoLookupService#findBroker()` will never complete. ### Modifications Limit the number of times a lookup is redirected to 100. This maximum is user configurable. If the number of redirects exceeds 100, the lookup will fail. But `ConnectionHandler` retries lookup so that the producer can eventually reconnect to the new broker.
This was referenced Jun 8, 2020
Huanli-Meng
pushed a commit
to Huanli-Meng/pulsar
that referenced
this pull request
Jun 12, 2020
Master Issue: apache#7041 ### Motivation When a leader broker is restarted, some producers for topics owned by that broker may not be reopened on the new broker. When this happens, message publishing will continue to fail until the client application is restarted. As a result of the investigation, I found that lookup requests sent by the producers in question are redirected more than 10,000 times between multiple brokers. When a lookup request is redirected, `BinaryProtoLookupService#findBroker()` is called recursively. Therefore, tens of thousands of redirects will cause `StackOverflowError` and `BinaryProtoLookupService#findBroker()` will never complete. ### Modifications Limit the number of times a lookup is redirected to 100. This maximum is user configurable. If the number of redirects exceeds 100, the lookup will fail. But `ConnectionHandler` retries lookup so that the producer can eventually reconnect to the new broker.
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
Master Issue: apache#7041 ### Motivation When a leader broker is restarted, some producers for topics owned by that broker may not be reopened on the new broker. When this happens, message publishing will continue to fail until the client application is restarted. As a result of the investigation, I found that lookup requests sent by the producers in question are redirected more than 10,000 times between multiple brokers. When a lookup request is redirected, `BinaryProtoLookupService#findBroker()` is called recursively. Therefore, tens of thousands of redirects will cause `StackOverflowError` and `BinaryProtoLookupService#findBroker()` will never complete. ### Modifications Limit the number of times a lookup is redirected to 100. This maximum is user configurable. If the number of redirects exceeds 100, the lookup will fail. But `ConnectionHandler` retries lookup so that the producer can eventually reconnect to the new broker.
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Nov 29, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Nov 29, 2022
5 tasks
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Nov 29, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 2, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 2, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 2, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 7, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 7, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 7, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 12, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 12, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 12, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 16, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 16, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 16, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Dec 16, 2022
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 3, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 3, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 3, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 3, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 4, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Jan 4, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
tongsucn
added a commit
to tongsucn/pulsar-client-cpp
that referenced
this pull request
Feb 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master Issue: #7041
Motivation
When a leader broker is restarted, some producers for topics owned by that broker may not be reopened on the new broker. When this happens, message publishing will continue to fail until the client application is restarted.
As a result of the investigation, I found that lookup requests sent by the producers in question are redirected more than 10,000 times between multiple brokers.
When a lookup request is redirected,
BinaryProtoLookupService#findBroker()is called recursively. Therefore, tens of thousands of redirects will causeStackOverflowErrorandBinaryProtoLookupService#findBroker()will never complete.Modifications
Limit the number of times a lookup is redirected to 100. This maximum is user configurable. If the number of redirects exceeds 100, the lookup will fail. But
ConnectionHandlerretries lookup so that the producer can eventually reconnect to the new broker.