Huge Off-heap memory used by java.nio.DirectByteBuffers

[dmks]

What version of gRPC are you using?

io.grpc:grpc-netty:1.20.0
io.grpc:grpc-stub:1.20.0
io.netty:netty-transport-native-epoll:4.1.34.Final:linux-x86_64
group: 'com.google.flatbuffers', name: 'flatbuffers-java', version: '1.10.0'
group: 'com.google.flatbuffers', name: 'flatbuffers-java-grpc', version: '1.10.0'

What did you expect to see?

No memory leak

Problem:

When I run stress tests with my grpc-java server on amazon linux 2, I saw huge process memory usage, but It was not heap memory or Metaspace (they are pretty small). so I took heap dump and use jxray to generate the report, I saw huge off-heap memory usage.

Here is the screenshot:

However, the server is running perfectly ok on my Mac pro with the same code.
And here is my code to start grpc server:
`ThreadFactory bossTF = new DefaultThreadFactory("boss-netty-", true /*daemon */);
ThreadFactory workerTF = new DefaultThreadFactory("worker-netty-", true /*daemon */);
final EventLoopGroup boss;
final EventLoopGroup worker;
final Class<? extends ServerChannel> channelType;

switch (transport) {
  case "nio": {
    boss = new NioEventLoopGroup(1, bossTF);
    worker = new NioEventLoopGroup(0, workerTF);
    channelType = NioServerSocketChannel.class;
    break;
  }
  case "epoll": {
    try {
      // These classes are only available on linux.
      Class<?> groupClass = Class.forName("io.netty.channel.epoll.EpollEventLoopGroup");
      @SuppressWarnings("unchecked")
      Class<? extends ServerChannel> channelClass = (Class<? extends ServerChannel>)
          Class.forName("io.netty.channel.epoll.EpollServerSocketChannel");
      boss =
          (EventLoopGroup)
              (groupClass
                  .getConstructor(int.class, ThreadFactory.class)
                  .newInstance(1, bossTF));
      worker =
          (EventLoopGroup)
              (groupClass
                  .getConstructor(int.class, ThreadFactory.class)
                  .newInstance(0, workerTF));
      channelType = channelClass;
      break;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
  default: {
    // Should never get here.
    throw new IllegalArgumentException("Unsupported transport: " + transport);
  }
}

server = NettyServerBuilder.forPort(port)
    .bossEventLoopGroup(boss)
    .workerEventLoopGroup(worker)
    .channelType(channelType)
    .addService(engine)
    .build()
    .start();

logger.info("Server started, listening on " + port);

`

Thanks

[carl-mastrangelo]

This is not a bug, it is how Netty managed memory. You can control it using the System properties found here:

https://github.com/netty/netty/blob/netty-4.1.34.Final/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L114

[BlueStalker]

@carl-mastrangelo But why above code runs well on Macbook? Is local Macbook has different settings on PooledByeBufAllocator?

[carl-mastrangelo]

@BlueStalker Can you turn on FINE or DEBUG level logging when you launch your application? This will tell what's going on with the initial memory pools. It should be the logs for the io.netty package.

[BlueStalker]

@carl-mastrangelo How can I enable the logging from grpc and netty with DEBUG level?
Currently our application is using log4j2, in our build.gradle file we have:
compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.11.2'
compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.11.2'

And in our log4j2.xml I inserted stuffs like:
`

  <AppenderRef ref="console" level="DEBUG" />

</Root>

<Logger name="io.netty" level="DEBUG" additivity="false">

  <AppenderRef ref="console" level="DEBUG" />

</Logger>

<Logger name="io.grpc" level="DEBUG" additivity="false">

  <AppenderRef ref="console" level="DEBUG" />

</Logger>

`

But It seemed like only logs coming from our application is showing.

[carl-mastrangelo]

Sorry, I don't know how to use Log4J. The behavior of high off heap allocation is completely normal. If it grows without bound that would be indicative of a bug.