Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Broker] Fix direct memory leak in getLastMessageId #10977

Merged

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 18, 2021

Fixes #7304

Motivation

See #7304 for the details. There's a memory leak when calling lastMessageId API.

I used the instructions by @megfigura in #7304 (comment) to reproduce.

In one terminal window I started pulsar-perf produce to create 5 msg/second to the topic called "test"

pulsar-perf produce -r 5 -m 100000 test

Then in another window I ran the loop to call lastMessageId in a tight loop:

while true; do curl http://pulsar-testenv-deployment-broker.pulsar-testenv.svc.cluster.local:8080/admin/v2/persistent/public/default/test/lastMessageId; echo ; done

In a third terminal window, I had the broker logs filtered for errors or leak logs :

kubectl -n pulsar-testenv logs -f pod/pulsar-testenv-deployment-broker-0 |tee broker_logs_`date +%s`.txt | egrep 'ERROR|OOM|LEAK'

The test environment was deployed with these values.yaml settings: https://github.com/lhotari/pulsar-playground/blob/f39b83b9e08f80af57fb70b1439f6c466c1f4405/test-env/1node/values.yaml

The memory leak problem reproduces almost immediately.

Modifications

Add the missing entry.release() call to PersistentTopic.getLastMessageId

@lhotari lhotari self-assigned this Jun 18, 2021
@lhotari lhotari added the type/bug The PR fixed a bug or issue reported a bug label Jun 18, 2021
@lhotari lhotari added this to the 2.9.0 milestone Jun 18, 2021
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui PTAL

@merlimat merlimat merged commit 7417ca8 into apache:master Jun 19, 2021
eolivelli pushed a commit to datastax/pulsar that referenced this pull request Jun 20, 2021
lhotari added a commit that referenced this pull request Jun 21, 2021
@lhotari lhotari added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Jun 21, 2021
lhotari added a commit that referenced this pull request Jun 21, 2021
@lhotari lhotari added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Jun 21, 2021
lhotari added a commit to datastax/pulsar that referenced this pull request Jun 21, 2021
yangl pushed a commit to yangl/pulsar that referenced this pull request Jun 23, 2021
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-picked/branch-2.7 Archived: 2.7 is end of life cherry-picked/branch-2.8 Archived: 2.8 is end of life release/2.7.3 release/2.8.1 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory leak on broker with streaming Spark application
5 participants