Endless Topic HTTP Request Redirects

[rfmyuk]

Describe the bug
Admin REST HTTP Request on topic may redirect multiple times to different broker endlessly.

This bug occurs randomly and at inconsistent times.

(The HTTP request must be related to topic ownership, e.g. get topic subscription, stats, etc...)

To Reproduce
Steps to reproduce the behavior:

1. Setup a Pulsar Cluster with 3 BookKeepers, 3 Global ZooKeepers, 3 ZooKeepers and 3 Brokers, and add JWT Authentication.
2. Open Postman (or any HTTP Client) and disable "Automatically follow redirects" in Settings.
3. Have the cluster up for a while (can be hours/days).
4. Send an HTTP request to create a topic.
5. If you get a redirect, follow the redirect (send the request to the IP in the "Location" header).
6. Keep following the redirects and you never get a response.

I used this broker.conf.

If you enable automatically follow redirects, the request will eventually return unauthorized 401 even though I provide a valid JWT that works with other non-topic related requests.

Shutting down 2 brokers (out of 3), inevitbly resolves the issue since there would be only one topic owner.

Expected behavior
I expect the redirects to eventually give me a broker that has the response for my request.

Desktop (please complete the following information):

• OS: CentOS 7

Additional context
Pulsar Version: 2.8.2 - 2.8.3 (may affect more versions, not tested)

As requested by @lhotari
jstack dumps:
Bugged when 3 brokers are up:

• Broker 1 - when it's bugged
• Broker 2 - when it's bugged
• Broker 3 - when it's bugged

Not bugged when 1 broker is up:

• Broker 1 - when it works

[mattisonchao]

I think i can help with it.

[rfmyuk]

@mattisonchao Are there any updates related to this issue?

[lhotari]

@ronfarkash Do you use the Pulsar Proxy?

[rfmyuk]

@ronfarkash Do you use the Pulsar Proxy?

No, 3 brokers and I send requests directly to them.

[rfmyuk]

@mattisonchao Asking again if there is anything in the works for this issue?

[eolivelli]

It it not clear to me which API has the problem.
Do we have a simple reproducer?

[eolivelli]

Which pulsar version?

[rfmyuk]

It it not clear to me which API has the problem. Do we have a simple reproducer?

Admin Rest API v2, as stated in the issue, if the HTTP request is related to topic ownership, it can start happening, and when it does start happening, a broker reset is required.

Which pulsar version?

Stated above, 2.8.2

[eolivelli]

Thank you very much for your answers.
Do you see loads in the broker about bundle unloading or bundle splitting related to that namespace?

There were (and probably still they are) problems with unloading. Last week I was investigating some of these problems in case of OOM during bundle unloading.

[rfmyuk]

Thank you very much for your answers. Do you see loads in the broker about bundle unloading or bundle splitting related to that namespace?

There were (and probably still they are) problems with unloading. Last week I was investigating some of these problems in case of OOM during bundle unloading.

I am not entirely sure if bundle unloading/splitting is the cause of the issue since I have not noticed any OOM problems, it almost seems random.

To my knowledge, there are multiple topic metadata & ownership issues that were being worked on, but sadly I am unaware of their status and have a hard time catching up due to busy schedule.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[mattisonchao]

@ronfarkash
Sorry for the late reply, I will check it and try to reproduce it.