Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LEAK: ByteBuf.release() was not called before it's garbage-collected. #3029

Closed
cobar79 opened this issue Jan 15, 2024 · 4 comments
Closed

LEAK: ByteBuf.release() was not called before it's garbage-collected. #3029

cobar79 opened this issue Jan 15, 2024 · 4 comments
Assignees
@violetagg
Labels
status/invalid We don't feel this issue is valid

Comments

@cobar79
Copy link

cobar79 commented Jan 15, 2024 •
edited
Loading 

    public Optional<Path> downloadAndUnZip(String url) throws MyClientException, MyFileException {
        String baseUrl =url;
        if (baseUrl == null) {
            throw new MyClientException("Invalid URL");
        }
        String uri = url.substring(url.indexOf(baseUrl) + baseUrl.length());
        log.info("Download uri: {}", uri);
        Mono<DataBuffer> dataBufferFlux = fetchDownloadAndUnZip(uri);
        dataBufferFlux.block();
        String exportZipName = uri.substring(uri.lastIndexOf("/"));
        return fileService.writeZipFile(dataBufferFlux, exportZipName);
    }

    private Mono<DataBuffer> fetchDownloadAndUnZip(String uri) {
        return myWebClient.get()
            .uri(uriBuilder -> uriBuilder
                .path(Objects.requireNonNull(uri))
                .build()
            )
            .accept(MediaType.APPLICATION_OCTET_STREAM)
            .retrieve()
            .onStatus(status -> status.value() == HttpStatus.NOT_FOUND.value(), clientResponse -> {
                log.error(uri + " " + HttpStatus.NOT_FOUND + " Error");
                return Mono.error(new MyClientException(HttpStatus.NOT_FOUND.value() + ":: " + uri + " NOT FOUND"));
            })
            .bodyToMono(DataBuffer.class)
            .retryWhen(Retry.backoff(myConfig.getDownloadRetryMax(), Duration.ofSeconds(myConfig.getDownloadWaitMillis()))
                .filter(MyClientException.class::isInstance)
                .onRetryExhaustedThrow((retryBackoffSpec, retrySignal) -> {
                    String message = HttpStatus.NOT_FOUND.value() + ":: " + uri + " NOT FOUND. Failed to download after " + myConfig.getDownloadRetryMax() + " retries";
                    throw new MyClientException(message);
                })
            )
            .doFinally(signalType -> {
                log.warn("Operation completed with signal type: " + signalType);

                // Cleanup logic or additional actions can be performed here

                // For example, if you need to log or perform some action based on success or error:
                if (signalType == SignalType.ON_COMPLETE) {
                    log.info("Operation completed successfully");
                } else if (signalType == SignalType.ON_ERROR) {
                    log.error("Operation completed with an error");
                }
            });
    }

Writing the Zip file:

    public Optional<Path> writeZipFile(Mono<DataBuffer> dataBuffer, String file) throws MyFileException {
        Path tempFilePath = Paths.get(getDownloadPath().toAbsolutePath() + File.separator + file);
        DataBufferUtils.write(dataBuffer, tempFilePath, StandardOpenOption.CREATE).block();
        log.debug("Zip file @ {}", tempFilePath.toAbsolutePath());
        return unzipUpdates(tempFilePath);
    }

Logs:

2024-15-01 12:16:01.060 [scheduling-1] INFO  com.dp.ingest.client.MyClientServiceImpl.downloadAndUnZip - Download uri: /20240115191500.export.CSV.zip

2024-15-01 12:16:01.505 [reactor-http-nio-2] INFO  com.dp.ingest.config.rest.MyWebClientConfig.lambda$logResponse$6 - Response: 200 OK

2024-15-01 12:16:01.507 [reactor-http-nio-2] WARN  com.dp.ingest.client.MyClientServiceImpl.lambda$fetchDownloadAndUnZip$7 - Operation completed with signal type: onComplete

2024-15-01 12:16:01.518 [reactor-http-nio-2] INFO  com.dp.ingest.client.MyClientServiceImpl.lambda$fetchDownloadAndUnZip$7 - Operation completed successfully

2024-15-01 12:16:01.556 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.writeZipFile - Zip file @ C:\projects\os\ingest\app\.\temp\20240115191500.export.CSV.zip

2024-15-01 12:16:01.567 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.unzipUpdates - Entry 20240115191500.export.CSV

2024-15-01 12:16:01.568 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.unzipUpdates - ENTRY AT .\temp\20240115191500.export.CSV

2024-15-01 12:16:01.572 [scheduling-1] INFO  com.dp.ingest.service.impl.MyIngestServiceImpl.processExportUpdates - Parsing .\temp\20240115191500.export.CSV, length = 524691

2024-15-01 12:16:02.129 [scheduling-1] INFO  com.dp.ingest.service.impl.PublishServiceImpl.queueEvents - Queueing 1302 events.

2024-15-01 12:18:47.949 [scheduling-1] INFO  com.dp.ingest.service.impl.PublishServiceImpl.queueEvents - Queued 1302 events to parse-events

2024-15-01 12:18:47.951 [scheduling-1] INFO  com.dp.ingest.service.impl.MyIngestServiceImpl.processExportUpdates - *** Processed 1302 events in 166891 millis.

2024-15-01 12:18:47.955 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.deleteTsvFile - DELETED:: .\temp\20240115191500.export.CSV

2024-15-01 12:18:47.956 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.deleteZipFile - delete url /20240115191500.export.CSV.zip

2024-15-01 12:18:47.960 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.deleteZipFile - DELETED:: /20240115191500.export.CSV.zip

2024-15-01 12:18:47.963 [scheduling-1] INFO  com.dp.ingest.service.impl.MyJob.executeJob - Finished Job in 167954 millis

2024-15-01 12:20:01.485 [reactor-http-nio-2] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1063)
        io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:656)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:285)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.indexOf(AdvancedLeakAwareByteBuf.java:647)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1051)
        io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:656)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:285)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#4:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1063)
        io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:1113)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:270)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#5:
        io.netty.buffer.AdvancedLeakAwareByteBuf.indexOf(AdvancedLeakAwareByteBuf.java:647)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1051)
        io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:1113)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:270)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#6:
        Hint: 'reactor.left.httpCodec' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#7:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#8:
        io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:635)
        io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
: 1 leak records were discarded because they were duplicates
: 41 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Note the signal type logs from the WebClient finally block.

Expected Behavior

More indication where the exception occurred.

Actual Behavior

Dumps a bunch of logs with no indication where in the application code the ByteBuf was created.

Steps to Reproduce

downloadAnUnzip("http://localhost/export.CSV.zip")

Possible Solution

Your Environment

Running with Spring Boot v3.1.4, Spring v6.0.12

  • Reactor version(s) used: 1.1.11
  • Other relevant libraries versions (eg. netty, ...): 4.1.97
  • JVM version (java -version): 17
  • OS and version (eg. uname -a): MINGW64_NT-10.0-19044
@cobar79 cobar79 added status/need-triage A new issue that still need to be evaluated as a whole type/bug A general bug labels Jan 15, 2024
@cobar79
Copy link
Author

cobar79 commented Jan 16, 2024

Investigating if the exception is coming from AWS SQS client 2.20.63

@violetagg
Copy link
Member

@cobar79 How to debug memory leaks please follow this FAQ

@violetagg
Copy link
Member

@cobar79 This code is interesting

public Optional<Path> downloadAndUnZip(String url) throws MyClientException, MyFileException {
        String baseUrl =url;
        if (baseUrl == null) {
            throw new MyClientException("Invalid URL");
        }
        String uri = url.substring(url.indexOf(baseUrl) + baseUrl.length());
        log.info("Download uri: {}", uri);
        Mono<DataBuffer> dataBufferFlux = fetchDownloadAndUnZip(uri);
-->  dataBufferFlux.block();
        String exportZipName = uri.substring(uri.lastIndexOf("/"));
        return fileService.writeZipFile(dataBufferFlux, exportZipName);
    }

First you block which will retrieve the DataBuffer and then you again try to process dataBufferFlux, I might be missing something here?

@violetagg violetagg added for/user-attention This issue needs user attention (feedback, rework, etc...) and removed status/need-triage A new issue that still need to be evaluated as a whole labels Jan 16, 2024
@violetagg violetagg self-assigned this Jan 16, 2024
@cobar79
Copy link
Author

cobar79 commented Jan 16, 2024

@violetagg Thank you for the observation. I missed the block on the DataBufferUtils.write.

@cobar79 cobar79 closed this as completed Jan 16, 2024
@violetagg violetagg added status/invalid We don't feel this issue is valid and removed type/bug A general bug for/user-attention This issue needs user attention (feedback, rework, etc...) labels Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@violetagg violetagg

Labels
status/invalid We don't feel this issue is valid
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@violetagg @cobar79