org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144

[yinjianfei]

Expected behavior

works fine

Actual behavior

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
	at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101) ~[spring-core-5.3.1.jar:5.3.1]
	at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94) ~[spring-core-5.3.1.jar:5.3.1]
	at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59) ~[spring-core-5.3.1.jar:5.3.1]
	at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:118) ~[reactor-core-3.4.0.jar:3.4.0]
	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.0.jar:3.4.0]
	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
	at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:346) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:535) ~[reactor-netty-http-1.0.1.jar:1.0.1]
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:229) ~[reactor-netty-http-1.0.1.jar:1.0.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]

2020-12-09 18:18:45.283 ERROR 20635 --- [or-http-epoll-3] r.n.channel.ChannelOperationsHandler     : [id: 0x2c8b7980, L:/0:0:0:0:0:0:0:1%0:7800 - R:/0:0:0:0:0:0:0:1%0:55390] Error was received while reading the incoming data. The connection will be closed.

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-buffer-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92) ~[netty-codec-http-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
	at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:349) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:535) ~[reactor-netty-http-1.0.1.jar:1.0.1]
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.1.jar:1.0.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:229) [reactor-netty-http-1.0.1.jar:1.0.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) [netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) [netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_275]

Steps to reproduce

post /memory/test with request body larger than 262144
using curl in: https://github.com/yinjianfei/spring-max-in-memory-test/blob/main/README.md

Reproducable code

@Slf4j
@Configuration
public class RouterConfig {
    @Bean
    public RouterFunction<ServerResponse> route() {
        return RouterFunctions.route()
                .POST("/memory/test", this::handle)
                .build();
    }

    private Mono<ServerResponse> handle(ServerRequest serverRequest) {
        return serverRequest.bodyToMono(JsonNode.class)
                .flatMap(body ->
                        ServerResponse.ok().syncBody(JsonNodeFactory.instance.objectNode())
                ).onErrorResume(ex -> {
                    log.error("error ", ex);
                    return ServerResponse.ok().syncBody(ex.getMessage());
                });
    }
}

application config

server:
  port: 7800
spring:
  codec:
    max-in-memory-size: 100MB

code repo at: https://github.com/yinjianfei/spring-max-in-memory-test

java --version

openjdk version "1.8.0_275"

springboot version

2.4.0

upper code work fine in release 2.1.4.release

[wilkinsona]

Thanks for the sample. You've used @EnableWebFlux which switches off Spring Boot's auto-configuration of WebFlux, including its customisation of the max in-memory buffer size. As a result, the spring.codec.max-in-memory-size property has no effect. The sample works for me once @EnableWebFlux has been removed.

[yinjianfei]

@wilkinsona Thanks
What should i do to make spring.codec.max-in-memory-size work along with webflux?

[yinjianfei]

it works ,thanks again

@Configuration
public class Configure implements WebFluxConfigurer {

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(10 * 1024 * 1024);
    }

}

[xa-yifan]

@yinjianfei 没有用

[cangkuren]

How about WebTestClient. When I do it in integration test, server side return 200, but client side throw this error
Objects.requireNonNull(webTestClient .post() .uri("/content-repo/api/v1/activities/search-latest") .contentType(MediaType.APPLICATION_JSON) .accept(MediaType.APPLICATION_JSON) .bodyValue(contentSearchCriteria) .exchange() .expectStatus() .is2xxSuccessful() .expectBody() .returnResult() .getResponseBody())

[wilkinsona]

@cangkuren If you use the auto-configured WebTestClient its max in memory size should be configured by the spring.codec.max-in-memory-size property. If you're creating your own, then you should configure the max in memory size via the builders. If you have any further questions, please follow up on Stack Overflow or Gitter. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

[cangkuren]

@cangkuren If you use the auto-configured WebTestClient its max in memory size should be configured by the spring.codec.max-in-memory-size property. If you're creating your own, then you should configure the max in memory size via the builders. If you have any further questions, please follow up on Stack Overflow or Gitter. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

I configure the max_in_memory_size in the application.yml, however it didn't work.

[wilkinsona]

This is now being discussed on Gitter.

[seetbw]

when you want read a request,there have a default max-in-memory-size,for fix this problem,you can set max-in-memory-size before you use DataBuffer

List<HttpMessageReader<?>> MESSAGE_READERS = HandlerStrategies.builder()
            .codecs(s -> s.defaultCodecs().maxInMemorySize(2 * 1024 * 1024))
            .build().messageReaders();
ServerRequest.create(exchange, MESSAGE_READERS);