Resteasy-reactive runs out of direct buffer memory when processing large multipart messages #26187
Comments
fedinskiy
changed the title
|
/cc @FroMage, @geoand, @stuartwdouglas |
|
@stuartwdouglas mind taking a look at this one? |
|
This issue is due to the usage of DirectByteBuffer in Netty. Having said this, I'm not sure how to approach this issue: If there is nothing we can do to tune Vert.x/Netty for a general use case (not only large files), I would say we should go for option a. |
|
Thanks a lot! I'll have to check it out |
|
Any idea what is allocating the large buffers? This sounds like a bug, we should not be allocating large buffers. |
|
To be honest, I'm not completely satisfied with the explanation, but haven't had time to look into it. |
|
How would I pass JVM args or system properties to the launched application with the testing framework in use? |
|
You can pass system properties by adding them here: https://github.com/quarkus-qe/quarkus-test-suite/blob/1f9a89d20514c269b4e87777dfb7dc62ef613f6b/http/rest-client-reactive/src/test/java/io/quarkus/ts/http/restclient/reactive/LargeFileHandlingIT.java#L50 But JVM args are not supported. Though ping me if you really need it and I will help here. |
|
Thanks |
|
I was not able to reproduce this under any combination of JVM options I tried. |
|
Closing this as we could not reproduce it. If you have newer information to provide, we will certainly take another look |
|
@geoand what newer information you need to reproduce this bug? |
|
I had tried multiple times to reproduce but could not |
|
quarkus-qe/quarkus-test-suite#816 was created ~1 hour ago, so the issue is still present. @geoand did you try to reproduce the issue using the reproducer provided by Fedor or did you use custom app/test? |
|
I'll reopen but if I can't reproduce it there is not much I can do |
I was using the reproducer |
|
@geoand how did you recreate memory-limited environment? The reproducer doesn't work on my machine, that why I had to put it into GH Actions. |
|
I don't remember. I'll have to retry it next week |
|
I can't reproduce this. I propose creating a reproducer using that standard Quarkus testing facilities - that way we will be able to try various things. |
|
@geoand I updated the linked reproducer[1] so it doesn't use test framework(see module "reproducer"). Also, I have an important addition: the problem does not appear, when |
|
Thanks for the update. I'll have a look soon |
|
I was still not able to reproduce the problem, but I'll have a closer look tomorrow. |
|
I was finally able to reproduce it |
|
This is a pretty weird issue - I was able to track down the root cause (I'll write it up later). Now I need to see how it can be addressed |
|
Vert.x component issue - eclipse-vertx/vert.x#4473 |
Prevent possible memory leak from Reactive REST Client multipart upload usage
…ad usage Fixes: quarkusio#26187 (cherry picked from commit c08560a)
Describe the bug
I have two Rest Reactive methods, one accepts File and another multipart of File and String. If running on resource limit machine(this behavior first arose on Github Actions), the second method fails with cryptic error while processing files of 2048 MB size, while the first handles larger(2054 MB) files without problems.
Expected behavior
Method should fail with more concise error or not fail at all.
Actual behavior
Method fails with this error:
How to Reproduce?
This steps should be run on memory limited machine(preferably, VM)
git clone https://github.com/fedinskiy/quarkus-test-suite.git -b reproducer/multipart_oom testscd testsmvn -Dall-modules -pl http/rest-client-reactive/ clean verify -Dit.test=LargeFileHandlingITTest
uploadMultipartfails, testsuploadBigFileThroughClientanduploadThroughClientwork.Output of
uname -aorver4.18.0-372.9.1.el8.x86_64
Output of
java -versionopenjdk 11.0.13 2021-10-19 LTS(also temurin 11.0.15)
GraalVM version (if different from Java)
No response
Quarkus version or git rev
2.7.6, 2.9.1 and ca836996b345a6cad93a79a6d8600c5dd809ed92
Build tool (ie. output of
mvnw --versionorgradlew --version)3.8.3, 3.8.4 and 3.8.6
Additional information
CI logs:
https://github.com/quarkus-qe/quarkus-test-suite/runs/6919232274?check_suite_focus=true
On 4G VM, which is described in the
environmentsection, the test do not threw any errors, but hangs with warnings like these, but with different thread numbers:The text was updated successfully, but these errors were encountered: