Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timed out requests causes a memory leak in NettyMessagingService #8541

Closed
romansmirnov opened this issue Jan 6, 2022 · 0 comments · Fixed by #8550
Closed

Timed out requests causes a memory leak in NettyMessagingService #8541

romansmirnov opened this issue Jan 6, 2022 · 0 comments · Fixed by #8550
Assignees
@romansmirnov
Labels
kind/bug Categorizes an issue or PR as a bug support Marks an issue as related to a customer support request version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1

Comments

@romansmirnov
Copy link
Member

romansmirnov commented Jan 6, 2022 •
edited
Loading

Describe the bug

The NettyMessagingService keeps track of open futures for each request being sent. Typically, the open futures are cleared when a response is received or when the NettyMessagingService closes. But there might be cases in which the recipient of the request never responds. As a consequence, the corresponding opened future gets completed exceptionally but it is never removed from the array of open futures:

https://github.com/camunda-cloud/zeebe/blob/73e5c7be9f453e30b5aebc07aae322ee5f82b11e/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/NettyMessagingService.java#L225-L234

This may result in OOM, especially, when the broker is running for long.

Expected behavior
Whenever a request times out, it should be removed from the opened futures array. So that it does not end up in a growing memory/heap consumption.

Environment:

  • Zeebe Version: 1.3.0

related to https://jira.camunda.com/browse/SUPPORT-11966
related to #6878
@romansmirnov romansmirnov added kind/bug Categorizes an issue or PR as a bug support Marks an issue as related to a customer support request labels Jan 6, 2022
@romansmirnov romansmirnov self-assigned this Jan 6, 2022
@romansmirnov romansmirnov mentioned this issue Jan 6, 2022
Merged
9 tasks
@ghost ghost closed this as completed in 944aee0 Jan 7, 2022
This was referenced Jan 7, 2022
Merged
Merged
ghost pushed a commit that referenced this issue Jan 7, 2022
@romansmirnov
merge: #8553
994b45f 
8553: [Backport stable/1.2] fix(atomix/netty): remove open future when request times out r=oleschoenburg a=github-actions[bot]

# Description
Backport of #8550 to `stable/1.2`.

relates to #8541

Co-authored-by: Roman <roman.smirnov@camunda.com>
ghost pushed a commit that referenced this issue Jan 7, 2022
@romansmirnov
merge: #8554
0730107 
8554: [Backport stable/1.3] fix(atomix/netty): remove open future when request times out r=oleschoenburg a=github-actions[bot]

# Description
Backport of #8550 to `stable/1.3`.

relates to #8541

Co-authored-by: Roman <roman.smirnov@camunda.com>
@npepinpe npepinpe added the version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1 label Jan 17, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@romansmirnov romansmirnov

Labels
kind/bug Categorizes an issue or PR as a bug support Marks an issue as related to a customer support request version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(atomix/netty): remove open future when request times out camunda/camunda
4 participants
@npepinpe @romansmirnov @remcowesterhoud @pihme