Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ExchangeStrategies customizations in WebClient #23961

Closed
totof3110 opened this issue Nov 9, 2019 · 16 comments
Closed

Allow ExchangeStrategies customizations in WebClient #23961

totof3110 opened this issue Nov 9, 2019 · 16 comments

Comments

@totof3110
Copy link

@totof3110 totof3110 commented Nov 9, 2019

Seems caused by 89d053d / #23884.

After upgrading from Spring Boot 2.2.0 to 2.2.1, WebClient started throwing org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144 when calling a JSON REST API (that has a large response size).

From the linked documentation (https://github.com/spring-projects/spring-framework/blob/master/src/docs/asciidoc/web/webflux.adoc#limits), it's not clear what configuration should be changed to make our API calls work again.

Is this expected behavior?

Full stack trace:

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
	at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Body from POST <redacted> [DefaultClientResponse]
Stack trace:
		at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
		at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94)
		at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59)
		at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:119)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
		at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:213)
		at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:346)
		at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:348)
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:572)
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478)
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
		at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
		at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:834)
@bclozel

This comment has been minimized.

Copy link
Member

@bclozel bclozel commented Nov 10, 2019

If you are using Spring Boot and creating your WebClient using a WebClient.Builder configured by Spring Boot (you can get it injected), you can use the new configuration property spring.codec.max-in-memory-size, see spring-projects/spring-boot#18828.

Otherwise if you’re creating that client from scratch you need to configure the codecs like:

ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
                .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(1024 * 10)).build();
WebClient webClient = WebClient.builder().exchangeStrategies(exchangeStrategies).build();
@totof3110

This comment has been minimized.

Copy link
Author

@totof3110 totof3110 commented Nov 10, 2019

Thanks @bclozel . Since I use the autowired WebClient.Builder, I've tried setting spring.codec.max-in-memory-size to a large size (500MB) but am still getting that error.

We happen to be using Spring HATEOAS and I wonder if something there is overriding this configuration. In particular there's class org.springframework.hateoas.config.WebClientConfigurer which transforms the WebClients with:

webClient.mutate().exchangeStrategies(hypermediaExchangeStrategies()).build()

I've already noticed that this seems to mess up the ObjectMapper used in the WebClient and drop all the com.fasterxml.jackson.databind.Modules that were previously autoconfigured.

Could that be the reason?

@bclozel bclozel self-assigned this Nov 11, 2019
@bclozel bclozel added this to the 5.2.2 milestone Nov 11, 2019
@bclozel

This comment has been minimized.

Copy link
Member

@bclozel bclozel commented Nov 11, 2019

We need to improve the documentation for the client side of things. This should help non Spring Boot users and developers building their own WebClient instances.

Now for HATEOAS, we're probably missing an extension point to allow it to customize the codecs without replacing them - we should provide ClientCodecConfigurer with an additional method (like we did for ServerCodecConfigurer).

@bclozel bclozel changed the title "Exceeded limit on max bytes to buffer : 262144" with WebClient Allow ExchangeStrategies customizations in WebClient Nov 29, 2019
@bclozel bclozel closed this in b3020bc Nov 29, 2019
@totof3110

This comment has been minimized.

Copy link
Author

@totof3110 totof3110 commented Nov 30, 2019

Thanks @bclozel !

@jhoeller

This comment has been minimized.

Copy link
Contributor

@jhoeller jhoeller commented Nov 30, 2019

@bclozel Is it intentional that we're introducing an ExchangeStrategies.mutate() method which is deprecated from day one? Is this meant to be a hint for other API designers?

@bclozel

This comment has been minimized.

Copy link
Member

@bclozel bclozel commented Dec 1, 2019

@jhoeller yes this is intentional. I’ve added a new method on the WebClient.Builder that takes itself an ExchangeStrategies.Builder to customize those. It is hard to go back from ExchangeStrategies to its builder form because of the nature of the underlying infrastructure.

We’ve added clone and mutate methods to solve the current problem first without breaking the contract and we’ll remove all that in a future release.

I’ve added comments along those lines in the commit message itself.

@bclozel bclozel removed this from the 5.2.2 milestone Dec 2, 2019
@bclozel

This comment has been minimized.

Copy link
Member

@bclozel bclozel commented Dec 2, 2019

This change is breaking integrations with other projects. Reverting for now and rescheduling to another version.

@bclozel bclozel reopened this Dec 2, 2019
@rstoyanchev rstoyanchev added this to the 5.2.2 milestone Dec 2, 2019
rstoyanchev added a commit that referenced this issue Dec 2, 2019
rstoyanchev added a commit that referenced this issue Dec 2, 2019
bclozel added a commit that referenced this issue Dec 2, 2019
As a follow-up of gh-23961, this change provides a way for custom codecs
to align with the default codecs' behavior on common features like
buffer size limits and logging request details.

Closes gh-24118
Co-authored-by: Rossen Stoyanchev <rstoyanchev@pivotal.io>
bclozel added a commit that referenced this issue Dec 2, 2019
As a follow-up of gh-23961, this change provides a way for custom codecs
to align with the default codecs' behavior on common features like
buffer size limits and logging request details.

Closes gh-24119
Co-authored-by: Rossen Stoyanchev <rstoyanchev@pivotal.io>
pacphi added a commit to pacphi/cf-butler that referenced this issue Dec 10, 2019
…naged version dependencies

* Fix issue with H2 by updating version to r2dbc-h2-0.8.1.RELEASE
* Adjust WebClientConfig.java as per spring-projects/spring-framework#23961 (comment) and increase spring.codec.max-in-memory-size property value in application.yml to 512000000
* Adjust SpaceUsersTask.java to flatMap on save
@chrisatrotter

This comment has been minimized.

Copy link

@chrisatrotter chrisatrotter commented Feb 27, 2020

Hi,
This is most likely the wrong place to ask such a question, but on the topic of customising the memory size I was wondering how would you do a unit test, boundary test, to check that the webclient does a request when it is within the memory limit and breaks when the requests exceeds the memory limit?

@rstoyanchev

This comment has been minimized.

Copy link
Contributor

@rstoyanchev rstoyanchev commented Feb 28, 2020

We have such tests for each codec (e.g. for Jackson) so you shouldn't have to test that. All you should test is how your app is configured. That aside you can probably use WebTestClient without a server and pass the exact chunks.

@chrisatrotter

This comment has been minimized.

Copy link

@chrisatrotter chrisatrotter commented Feb 28, 2020

@rstoyanchev, you're right. I just wanted to create a simple unit test to ensure that the the custom memory size was set correctly. This is a snippet of the solution I managed to do with some Kotlin code:

@SpringBootTest
class MyTest(@Value("100") private val maxMemorySize: Int) {
     ...
@Test
    fun `DataBufferLimitException is thrown when 'memoryBufferData' is greater than the custom 'maxMemorySize (100)'`() {
        // Given:
        val memoryBufferData = "A".repeat(maxMemorySize)
        val toJson = "\"$memoryBufferData\""
        val mockResponse = MockResponse().setBody(toJson)
        val webClient = MockWebServer()
                    .also { it.enqueue(mockResponse) }
                    .also { it.start() }

        // Then:
        assertThrows<DataBufferLimitException> {
            webClient.get(path = "/test", entityType = Any::class)
        }
    }

@Test
    fun `Response is the same as 'memoryBufferData' when memoryBuffer is within the custom 'maxMemorySize (100)'`() {
        // Given:
        val belowMaxMemorySize = maxMemorySize - 2
        val memoryBufferData = "A".repeat(belowMaxMemorySize)
        val toJson = "\"$memoryBufferData\""
        val mockResponse = MockResponse().setBody(toJson)
        val webClient = MockWebServer()
                    .also { it.enqueue(mockResponse) }
                    .also { it.start() }

        // When:
        val response = webClient.get(path = "/test", entityType = Any::class)

        // Then:
        assertEquals(response, memoryBufferData)
    ...
}

Thank you for taking the time @rstoyanchev !

@datumgeek

This comment has been minimized.

Copy link

@datumgeek datumgeek commented Mar 31, 2020

i was getting this error for a simple RestController (i post a large json string).

here is how i successfully changed the maxInMemorySize

import org.springframework.context.annotation.Configuration;
import org.springframework.http.codec.ServerCodecConfigurer;
import org.springframework.web.reactive.config.ResourceHandlerRegistry;
import org.springframework.web.reactive.config.WebFluxConfigurer;

@Configuration
public class WebfluxConfig implements WebFluxConfigurer {

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        
        registry.addResourceHandler("/swagger-ui.html**")
            .addResourceLocations("classpath:/META-INF/resources/");

        registry.addResourceHandler("/webjars/**")
            .addResourceLocations("classpath:/META-INF/resources/webjars/");
    }

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
    }
}

this was surprisingly hard to find

@bclozel

This comment has been minimized.

Copy link
Member

@bclozel bclozel commented Mar 31, 2020

@datumgeek we've got this covered in the reference docs. Don't hesitate to share improvement ideas!

Thanks!

@datumgeek

This comment has been minimized.

Copy link

@datumgeek datumgeek commented Mar 31, 2020

@bclozel - thank you for the reference !!

i did see that part of the doc... but from that description, i wasn't able to figure out how to fix the error in the rest controller... maybe it needs a reference to a code sample? i'm guessing this is a fairly common issue for folks developing rest controllers...

the answer was also not present in the stackoverflow questions i found when searching. i tried to update a few of them 😄

once you have the recipe, it is very easy 😉

@rstoyanchev

This comment has been minimized.

Copy link
Contributor

@rstoyanchev rstoyanchev commented Mar 31, 2020

@datumgeek

This comment has been minimized.

Copy link

@datumgeek datumgeek commented Mar 31, 2020

@rstoyanchev - maybe i'm just being dense 😉

i was trying to configure the maxInMemoryBytes for the rest controller

i was thinking it would be good if there was a code sample for how to do this via WebFluxConfigurer

like maybe if it said specifically, if you are trying to change the maxInMemoryBytes for a RestController do this:

import org.springframework.context.annotation.Configuration;
import org.springframework.http.codec.ServerCodecConfigurer;
import org.springframework.web.reactive.config.WebFluxConfigurer;

@Configuration
public class WebfluxConfig implements WebFluxConfigurer {

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
    }
}
rstoyanchev added a commit that referenced this issue Mar 31, 2020
@rstoyanchev

This comment has been minimized.

Copy link
Contributor

@rstoyanchev rstoyanchev commented Mar 31, 2020

No that's fine, I made an improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.