Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 4134: Fixing memory leak error when using DirectEntryLogger #4135

Merged
merged 4 commits into from
Jan 8, 2024

Conversation

graysonzeng
Copy link
Contributor

Descriptions of the changes in this PR:

Motivation

When we use DirectIO and a catch up read occurs, a memory leak occurs because the ByteBuf is not released.

Because although in the internalReadEntry method in DirectEntryLogger, even if the upper layer fillReadAheadCache uses the ReferenceCountUtil.release(entry) method in the finally block to release ByteBuf, it will still cause an exception thrown by internalReadEntry, and the entryLogger.readEntry method happens to be in try. ..catch unexpected, causing ByteBuf to not be released correctly

Changes

Master Issue: #4134

@graysonzeng
Copy link
Contributor Author

oom

After this repairing and observing for a day, the OOM problem no longer occurs in the direct memory.

Copy link
Member

@horizonzy horizonzy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question about this change.

The issue is fixed by line_255, right? That means the entry logger file already be dirty.

If the code reaches line_266, the buf should be null.

@zymap zymap self-requested a review November 30, 2023 05:32
@graysonzeng
Copy link
Contributor Author

I have a question about this change.

The issue is fixed by line_255, right? That means the entry logger file already be dirty.

If the code reaches line_266, the buf should be null.

Yes it has been fixed by line_255, should I remove line_266?

@horizonzy
Copy link
Member

I have a question about this change.
The issue is fixed by line_255, right? That means the entry logger file already be dirty.
If the code reaches line_266, the buf should be null.

Yes it has been fixed by line_255, should I remove line_266?

I think it can be removed.

@graysonzeng
Copy link
Contributor Author

I have a question about this change.
The issue is fixed by line_255, right? That means the entry logger file already be dirty.
If the code reaches line_266, the buf should be null.

Yes it has been fixed by line_255, should I remove line_266?

I think it can be removed.

line_266 has been removed, thanks~

@horizonzy
Copy link
Member

horizonzy commented Nov 30, 2023

As I said, although this PR could fix the memory leak, the entry log file already be dirty, it can't read data correctly.

There is an issue with the entry log file about dirty. #4040

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch !

Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to move the ByteBuf out of the try-catch code block.

@graysonzeng
Copy link
Contributor Author

We don't need to move the ByteBuf out of the try-catch code block.

Fixed it

@hangc0276 hangc0276 merged commit cc1c2e0 into apache:master Jan 8, 2024
16 checks passed
hangc0276 pushed a commit to hangc0276/bookkeeper that referenced this pull request Jan 18, 2024
…ache#4135)

### Motivation

When we use DirectIO and a catch up read occurs, a memory leak occurs because the ByteBuf is not released.

Because although in the `internalReadEntry` method in `DirectEntryLogger`, even if the upper layer `fillReadAheadCache` uses the `ReferenceCountUtil.release(entry)` method in the finally block to release ByteBuf, it will still cause an exception thrown by `internalReadEntry`, and the `entryLogger.readEntry` method happens to be in try. ..catch unexpected, causing ByteBuf to not be released correctly

### Changes

Master Issue: apache#4134

(cherry picked from commit cc1c2e0)
Ghatage pushed a commit to sijie/bookkeeper that referenced this pull request Jul 12, 2024
…ache#4135)

### Motivation

When we use DirectIO and a catch up read occurs, a memory leak occurs because the ByteBuf is not released. 

Because although in the `internalReadEntry` method in `DirectEntryLogger`, even if the upper layer `fillReadAheadCache` uses the `ReferenceCountUtil.release(entry)` method in the finally block to release ByteBuf, it will still cause an exception thrown by `internalReadEntry`, and the `entryLogger.readEntry` method happens to be in try. ..catch unexpected, causing ByteBuf to not be released correctly

### Changes

Master Issue: apache#4134
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants