Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extremely bizarre leak from EpollRecvByteAllocatorHandle? #9917

Closed
astei opened this issue Jan 1, 2020 · 1 comment
Closed

Extremely bizarre leak from EpollRecvByteAllocatorHandle? #9917

astei opened this issue Jan 1, 2020 · 1 comment

Comments

@astei
Copy link

astei commented Jan 1, 2020

Expected behavior

No memory leak, application is stable over time.

Actual behavior

Small memory leak that eventually causes our application to be OOMed by the kernel after 11 days of continous uptime. We have captured this leak, running our application with -Dio.netty.leakDetection.level=advanced:

[04:07:09 ERROR] [io.netty.util.ResourceLeakDetector]: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
        io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Unknown Source)

Steps to reproduce

  1. Start application
  2. Let it run for a few hours with player connections (this is a Minecraft proxy)
  3. Receive ResourceLeakDetector warning

Minimal yet complete reproducer code (or URL to code)

I don't have a minimal reproduction case, but the application is open source at https://github.com/VelocityPowered/Velocity/tree/decode-multiple

Netty version

4.1.43.Final

JVM version (e.g. java -version)

openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.17.0, JRE 11 Linux amd64-64-Bit Compressed References 20191016_358 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 2a7af5674b based on jdk-11.0.5+10)

OS version (e.g. uname -a)

The application runs inside a Docker container based on the adoptopenjdk/openjdk11-openj9:alpine-jre image. The container has the following uname -a output:

Linux proxy-date96 4.19.0-0.bpo.6-amd64 #1 SMP Debian 4.19.67-2~bpo9+1 (2019-09-10) x86_64 Linux

@astei
Copy link
Author

astei commented Jan 1, 2020

Apparently I do not have full information needed to get a full report to you all (mistakenly have logs only from the default simple level). Closing this for now and reopening when I have more information and I'm truly sure it's a Netty problem (I have not proven this yet).

@astei astei closed this as completed Jan 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant