Limit the client reconnect behavior.

[codelipenghui]

Actual behavior

I got some abnormal metrics in broker stats.

When this happened, I checked the network metrics. The network metrics showed network was saturated.

So, I searched the error log in pulsar cluster, and found some strange logs:

Next, i searched the client logs in kibana:

Regardless the client exception, I am concerned about the client reconnect behavior. I think client reconnect too much and load data from bookie to cause the high network bandwidth usage. And pulsar client exposes updateServiceUrl() to users, it will also trigger reconnect behavior.

So we should add some mechanisms to limit the reconnect to avoid high network bandwidth usage.

[sijie]

@codelipenghui are you working on a fix for this?

[codelipenghui]

@sijie Yes, i will fix it, We need to decide work on broker or client ?

[sijie]

@codelipenghui thank you, what is your thought around? can you describe both approaches and compare pros and cons?

[codelipenghui]

@sijie

I think more secure and unified on broker side, broker already limit the maxConsumersPerTopic and maxConsumersPerSubscription, broker limit the client reconnect(actually limit subscribe) is taken for granted.

If on client side, every client should implement this limiter.

But to identify consumer, on client side is more simple. On broker side, broker can get the consumerId, consumerName, ip address. We can use ip address + consumerName + consumerId to identify a consumer

We just add a subscribe limiter on broker/namespace level. When broker handle subscribe request should check the limiter.

When client got the limit error, client don't retry again.

Subscribe limiter config in broker.conf. limiter support 2 parameters of resetPeriod and maxSubscribePerConsumer.

On client side, it's necessary to add metrics record the re-connect times.

[codelipenghui]

PR #2977

[sijie]

This was fixed by #2977