Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP form-data OutOfDirectMemoryError #10424

Closed
wuxiansen opened this issue Jul 23, 2020 · 2 comments · Fixed by #10457
Closed

HTTP form-data OutOfDirectMemoryError #10424

wuxiansen opened this issue Jul 23, 2020 · 2 comments · Fixed by #10457

Comments

@wuxiansen
Copy link
Contributor

Expected behavior

release buffer

Actual behavior

no release

SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.nioBuffer(AdvancedLeakAwareByteBuf.java:712)
	io.netty.handler.codec.http.multipart.AbstractDiskHttpData.addContent(AbstractDiskHttpData.java:159)
	io.netty.handler.codec.http.multipart.DiskAttribute.addContent(DiskAttribute.java:133)
	io.netty.handler.codec.http.multipart.MixedAttribute.addContent(MixedAttribute.java:137)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.loadDataMultipartStandard(HttpPostMultipartRequestDecoder.java:1328)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.loadDataMultipart(HttpPostMultipartRequestDecoder.java:1344)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:544)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.findMultipartDisposition(HttpPostMultipartRequestDecoder.java:787)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:496)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.findMultipartDelimiter(HttpPostMultipartRequestDecoder.java:648)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:483)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.parseBodyMultipart(HttpPostMultipartRequestDecoder.java:448)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.parseBody(HttpPostMultipartRequestDecoder.java:417)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.offer(HttpPostMultipartRequestDecoder.java:342)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.<init>(HttpPostMultipartRequestDecoder.java:184)
	io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.<init>(HttpPostRequestDecoder.java:93)
	io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.<init>(HttpPostRequestDecoder.java:69)
	com.netty.http.example.handler.Request.<init>(Request.java:88)
	com.netty.http.example.handler.Request.build(Request.java:601)
	com.netty.http.example.handler.HttpRequestHandler.channelRead0(HttpRequestHandler.java:35)
	com.netty.http.example.handler.HttpRequestHandler.channelRead0(HttpRequestHandler.java:28)
	io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	java.lang.Thread.run(Thread.java:748)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1858)
	io.netty.buffer.CompositeByteBuf.copy(CompositeByteBuf.java:1510)
	io.netty.buffer.AbstractUnpooledSlicedByteBuf.copy(AbstractUnpooledSlicedByteBuf.java:224)
	io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1195)
	io.netty.buffer.WrappedByteBuf.copy(WrappedByteBuf.java:882)
	io.netty.buffer.AdvancedLeakAwareByteBuf.copy(AdvancedLeakAwareByteBuf.java:695)
	io.netty.handler.codec.http.multipart.AbstractDiskHttpData.addContent(AbstractDiskHttpData.java:159)
	io.netty.handler.codec.http.multipart.DiskAttribute.addContent(DiskAttribute.java:133)
	io.netty.handler.codec.http.multipart.MixedAttribute.addContent(MixedAttribute.java:137)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.loadDataMultipartStandard(HttpPostMultipartRequestDecoder.java:1328)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.loadDataMultipart(HttpPostMultipartRequestDecoder.java:1344)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:544)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.findMultipartDisposition(HttpPostMultipartRequestDecoder.java:787)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:496)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.findMultipartDelimiter(HttpPostMultipartRequestDecoder.java:648)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.decodeMultipart(HttpPostMultipartRequestDecoder.java:483)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.parseBodyMultipart(HttpPostMultipartRequestDecoder.java:448)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.parseBody(HttpPostMultipartRequestDecoder.java:417)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.offer(HttpPostMultipartRequestDecoder.java:342)
	io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.<init>(HttpPostMultipartRequestDecoder.java:184)
	io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.<init>(HttpPostRequestDecoder.java:93)
	io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.<init>(HttpPostRequestDecoder.java:69)
	com.netty.http.example.handler.Request.<init>(Request.java:88)
	com.netty.http.example.handler.Request.build(Request.java:601)
	com.netty.http.example.handler.HttpRequestHandler.channelRead0(HttpRequestHandler.java:35)
	com.netty.http.example.handler.HttpRequestHandler.channelRead0(HttpRequestHandler.java:28)
	io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	java.lang.Thread.run(Thread.java:748)

Steps to reproduce

Netty Main

	public static ChannelFuture start() { 
		final ServerBootstrap serverBootstrap = new ServerBootstrap();
		serverBootstrap.group(boss, worker)
			.option(ChannelOption.SO_BACKLOG, 1024)
			.channel(NioServerSocketChannel.class) 
			.childHandler(new HttpServerChannelInitializer());
 
		ChannelFuture channelFuture = serverBootstrap.bind(6606).syncUninterruptibly();
		channel = channelFuture.channel(); 
		Assert.isTrue(channelFuture.isSuccess(), "Netty server start fail");
		return channelFuture;
	}

	public static void main(String[] args) {
		try {

			NettyDirectReport report = new NettyDirectReport();
			report.start();
			ChannelFuture start = start();
			start.channel().closeFuture().syncUninterruptibly();
		} finally {
			Runtime.getRuntime().addShutdownHook(new Thread(NettyApp::destroy));
		}
	}

HttpServerChannelInitializer.java

public class HttpServerChannelInitializer extends ChannelInitializer<SocketChannel> {

    @Override
    protected void initChannel(SocketChannel channel) throws Exception {
        channel.pipeline()
                .addLast(new HttpServerCodec())
                .addLast(new StringDecoder(StandardCharsets.UTF_8))
                .addLast(new HttpObjectAggregator(65536))
                .addLast(new ChunkedWriteHandler())
                .addLast(new HttpRequestHandler());
    }
}

HttpRequestHandler.java

public class HttpRequestHandler extends SimpleChannelInboundHandler<FullHttpRequest> {

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest fullHttpRequest) throws Exception {

        // this get http context
        final Request request = Request.build(ctx, fullHttpRequest);
        final Response response = Response.build(ctx, request);
        try {
            if (log.isDebugEnabled()) {
                log.debug("remote {} request :{}", request.getIp(), request.getPath());
            }

            response.setContent(Jackson.object2json(R.success(RandomUtil.simpleUUID())));

            if (!response.isSent()) {
                response.send();
            }
        } catch (Exception e) {
            log.error("base error message", e);
            response.sendError(HttpResponseStatus.INTERNAL_SERVER_ERROR, R.fail(e.getMessage()));
        }
    }
}

Request.java

	private Request(ChannelHandlerContext ctx, FullHttpRequest nettyRequest) {
		this.nettyRequest = nettyRequest;
		this.path = UrlUtil.getPath(getUri());

		this.putHeadersAndCookies(nettyRequest.headers());

		// request URI parameters
		this.putParams(new QueryStringDecoder(nettyRequest.uri()));
		if (nettyRequest.method() != HttpMethod.GET) {
			HttpPostRequestDecoder decoder = null;
			try {
                                // handler content 
				decoder = new HttpPostRequestDecoder(HTTP_DATA_FACTORY, nettyRequest);
				decoder.setDiscardThreshold(0);
				this.putParams(decoder);
				body = bufToString(nettyRequest.content());
			} finally {
				if (null != decoder) {
					decoder.destroy();
				}
			}
		} else {
			body = null;
		}
		this.putIp(ctx);
	}

Minimal yet complete reproducer code (or URL to code)

AbstractDiskHttpData.java

ByteBuffer byteBuffer = buffer.nioBufferCount() == 1 ? buffer.nioBuffer() : buffer.copy().nioBuffer();

this byteBuffer no release, should byteBuffer.release() Or don't copy buffer

    @Override
    public void addContent(ByteBuf buffer, boolean last)
            throws IOException {
        if (buffer != null) {
            try {
                int localsize = buffer.readableBytes();
                checkSize(size + localsize);
                if (definedSize > 0 && definedSize < size + localsize) {
                    throw new IOException("Out of size: " + (size + localsize) +
                            " > " + definedSize);
                }
                // *this byteBuffer  no release*
                ByteBuffer byteBuffer = buffer.nioBufferCount() == 1 ? buffer.nioBuffer() : buffer.copy().nioBuffer();
                int written = 0;
                if (file == null) {
                    file = tempFile();
                }
                if (fileChannel == null) {
                    RandomAccessFile accessFile = new RandomAccessFile(file, "rw");
                    fileChannel = accessFile.getChannel();
                }
                while (written < localsize) {
                    written += fileChannel.write(byteBuffer);
                }
                size += localsize;
                buffer.readerIndex(buffer.readerIndex() + written);
            } finally {
                // Release the buffer as it was retained before and we not need a reference to it at all
                // See https://github.com/netty/netty/issues/1516
                buffer.release();
            }
        }
       ......
    }

Netty version

4.1.50.Final

JVM version (e.g. java -version)

java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

OS version (e.g. uname -a)

Windows
Linux
@Zeymo
Copy link

Zeymo commented Jul 23, 2020

@wuxiansen Fix by #10360

@wuxiansen
Copy link
Contributor Author

@wuxiansen Fix by #10360

👌

@wuxiansen wuxiansen mentioned this issue Jul 30, 2020
Closed
@wuxiansen wuxiansen mentioned this issue Aug 7, 2020
Merged
normanmaurer pushed a commit that referenced this issue Aug 13, 2020
@wuxiansen
Fix #10434 OutOfDirectMemoryError causes cpu load too high and socket…
54bfd21 
… is full (#10457)

Motivation:

When we were using the netty http protocol, OOM occurred, this problem has been in 4.1.51.Final Fix [# 10424](#10424), even if OOM is up, the service will still receive new connection events, will occur again OOM and eventually cause the connection not to be released. 

code `byteBuf = allocHandle.allocate(allocator);`

Modification:

I fail to create buffer when I try to receive new data, i determine if it is OOM then the close read event releases the connection.
```java
        if (close || cause instanceof OutOfMemoryError || cause instanceof IOException) {
            closeOnRead(pipeline);
        }
```

Result:

Fixes # [10434](#10434).
normanmaurer pushed a commit that referenced this issue Aug 13, 2020
@wuxiansen @normanmaurer
Fix #10434 OutOfDirectMemoryError causes cpu load too high and socket…
223422c 
… is full (#10457)

Motivation:

When we were using the netty http protocol, OOM occurred, this problem has been in 4.1.51.Final Fix [# 10424](#10424), even if OOM is up, the service will still receive new connection events, will occur again OOM and eventually cause the connection not to be released.

code `byteBuf = allocHandle.allocate(allocator);`

Modification:

I fail to create buffer when I try to receive new data, i determine if it is OOM then the close read event releases the connection.
```java
        if (close || cause instanceof OutOfMemoryError || cause instanceof IOException) {
            closeOnRead(pipeline);
        }
```

Result:

Fixes # [10434](#10434).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix #10434 OutOfDirectMemoryError causes cpu load too high and socket is full wuxiansen/netty
2 participants
@Zeymo @wuxiansen