-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OutOfDirectMemoryError when modify response body #1404
Comments
Can you please retry with boot 2.1.11 and spring cloud Greenwich.SR4 or boot 2.2.2 and cloud Hoxton.SR1? |
The production environment has this problem. It is difficult to retry. I checked the code of boot 2.1.11 and spring cloud Greenwich.SR4 and boot 2.2.2 and cloud Hoxton.SR1, but the code with problems is the same. |
@wangjunhua123 did you try with Hoxton or just look at the code? Changes were made in reactor and framework as well. So boot 2.2.6 and Hoxton.SR3 are the latest. |
@wangjunhua123 @spencergibb Any updates on this? We are also experiencing the gateway allocating Direct Memory indefinitely until this error occurs |
@wangjunhua123 @gihad did either of you try with boot 2.2.6 and Hoxton.SR3? Can anyone provide steps to reproduce it (size of response, how many requests over how long?) |
@spencergibb Yes, we bumped version to 2.2.6.RELEASE and Hoxton.SR3 a few days ago after seeing your comment. We saw some improved memory usage after that but still dealing with this issue. We recently deployed our spring-cloud-gateway based app that we call "Portal V2" which is the API Gateway for a service oriented architecture with a few dozens of different services behind it. It was made as a direct replacement to our previous API gateway which used Zuul 1, "Portal V1". We see the amount of direct memory usage (which by default is equal to the Java heap as per the Xmx setting) go up very fast after deploy to close to the MAX and than slowly continue up over time until it hits the max and those exceptions start to happen. The same thing happens when giving it very large memory limits, it just takes longer. We can see this behavior at around 650 req/second. The requests that go through this API Gateway can be very different from one another. Most are lightweight requests with small responses but some can have large bodies (e.g.: 460Kb gzipped). We have several filters and do all sorts of things from the gateway like request composition using the Webclient, hydration of requests, and so on. Here are some containers reaching max direct memory usage quickly, when they hit the top we start getting those exceptions and therefore requests start to fail: After tinkering with this for the last 3 days my hunch is that it's more to do with netty settings that need to be properly tweaked for our API Gateway usage patterns. Our team is still experimenting and trying to find the root cause. cc @ukayani We have a very large TV featuring for one of our apps coming up so we will be working on this a lot this week. Thanks. |
What percentage of those requests modify the response body? In all my talks when I talk about modifying bodies is that the number of concurrent requests modifying a body * size of the body the minimum amount of memory needed (likely more). |
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed. |
We were able to contain the growth of native memory to a much more reasonable level (now it grows at a slow pace and it even stabilizes). The relationship between native memory usage and number of concurrent requests is more direct and predictable and that's the main factor now in estimating how much memory it needs. The main 2 properties that made the different were:
We saw some of these values were being used as defaults in Twitter's finagle and just copied the to start. In practice (e.g.: avg latency) we didn't experience any performance issues after setting those properties. |
@spencergibb I tried to set the two properties that @gihad mentioned and -XX:MaxDirectMemorySize=256m,but it seemed not working.I tested with 1000 threads and got the below exception:
The direct memory kept increasing untill I stop the test but wont release. Spring Cloud Greenwich.SR2 |
@spencergibb Sorry, I changed my job, this issue will be followed up by my colleague in the future, I can also communicate together, thank you! |
update a leak detection log:
|
spring boot 2.2.6 `@Override
|
Sounds like the problem fixed in PR #1766. |
Thanks for the reply, but not very understanding, and what is the solution, can you give me an example? |
The "text" I got from the response last modified the response to "content". Isn't this a modification? |
@Xpenf its a modification, but It looks like you are not using standard response body modification filter: And the merge request referenced earlier is fixing a bug within that filter. So that is unlikely its the same bug |
Hi, Spring Cloud Hoxton.SR8 Gateway simple application
@scheduled(cron = "*/10 * * * * ?") } I monitor direct_ MEMORY_ Counter has been increasing io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023410183, max: 1037959168) |
Has it been resolved? |
Has it been resolved? my version: |
update: spring-cloud.version: Hoxton.SR9 I customize a filter to rewrite the request like this:
then I test with a POST request but body param is form-data.I can get error with 415 Unsupported Media Type and error below:
I keep testing with this kind of request,I find that the direct buffers keep increasing till java.lang.OutOfMemoryError: Direct buffer memory. prometheus JVM monitor dashboard: direct buffer memory error info:
|
update: I add some codes about Databuffer as below in my custom global exception handler.The issue seems to be improved.
|
We've added some fixes as well. If this still happens in 4.1.2 (releasing later this month May 2024) we can reopen. If you want to try 4.1.2-SNAPSHOT, go ahead. Sorry for the late reply. |
Hi,
I want to report a problem.
Spring Cloud Greenwich.SR2
Spring Boot 2.1.6.RELEASE
When modify response body, allocated 16777216 byte(s) direct memory , but it's not release,cause OutOfDirectMemoryError.
The test found that the code for increasing direct memory is as follows:
org.springframework.cloud.gateway.filter.factory.rewrite.ModifyResponseBodyGatewayFilterFactory:
Direct memory increase code:
org.springframework.http.codec.EncoderHttpMessageWriter:
The text was updated successfully, but these errors were encountered: