Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Add publish rate limit for each broker to avoid OOM #5710
Through #3985, user could set the publish rate for each topic, but the topic number for each broker is not limited, so there is case that a lot of topics served in same broker, and if each topic send too many message, it will cause the messages not able to send to BookKeeper in time, and messages be hold in the direct memory of broker, and cause Broker out of direct memory.
Verifying this change
unit test passed.
sijie left a comment
@jiazhai overall looks good.
Can you create a task to follow up a future work - we should consider automatically configuring the max bytes based on the NIC bandwidth that broker detects from the system. Pulsar already has this mechanism detecting the NIC bandwidth for load report. We should piggyback this functionality to re-use that feature.
@rdhabalia I just replied to your comment in #5513. so this change here is using the mechanism you added for namespace rate limiter. what it basically does is having a rate calculator at broker level and piggyback the rate limiting by checking if publish exceeds a certain rate threshold (both namespace setting and broker setting). It is a pretty simple change that piggyback to your mechanism. why is this a concern to you?