You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Tengine buffers the entire response body into RAM before sending it.
Ⅱ. Describe what happened
If serving a file, Tengine will read the entire file into RAM (visible from htop), and after that is done, it will send the file to the client.
Ⅲ. Describe what you expected to happen
Tengine should not buffer the entire file in RAM. It should stream the file as it is read from the disk. Memory usage should not increase significantly.
Ⅳ. How to reproduce it (as minimally and precisely as possible)
Compile Tengine with XQUIC support
Acquire QUIC capable HTTP client (I used docker image ymuski/curl-http3 for local testing, but Firefox also triggers this)
Configure Tengine to serve files from a directory
Create a large file, or truncate -s 2G 2.bin
curl --http3 -v -k https://localhost/2.bin. There is a substantial delay before curl begins to receive the response. It is visible from htop that nginx will rapidly consume memory until it buffers the entire file in RAM.
Ⅴ. Anything else we need to know?
It appears the issue is caused in part by ngx_http_xquic_send_chain reading the entire response into another chain. This prevents normal backpressure from happening, which results in the entire response being read into the buffer. The patch below makes it so that Tengine will no longer wait until the entire file is in RAM to begin sending the response. However, it does not fully fix the issue.
diff --git a/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c b/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c
index 6eff9254..8b0fb707 100644
--- a/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c+++ b/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c@@ -648,6 +648,9 @@ ngx_http_xquic_send_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
/* update h3_stream->output_queue here */
h3_stream = r->xqstream;
+ // HACK: stop accepting new buffers if we have been limited by flow control+ if (!h3_stream->wait_to_write) {+
last_chain = h3_stream->output_queue;
while (last_chain != NULL) {
@@ -705,10 +708,12 @@ ngx_http_xquic_send_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
r->xqstream->queued++; /* used to count buffers not sent*/
}
+ // end hack+ }
last_out = h3_stream->output_queue;
send = 0;
With the patch, Tengine will still consume the same amount of RAM, but it will also busy-wait on epoll_wait. I am not sure how to fix this.
Ⅵ. Environment:
Tengine version (use sbin/nginx -V): Commit 04baff4 on branch master (latest commit as of issue creation)
The text was updated successfully, but these errors were encountered:
iczero
changed the title
Tengine with XQUIC will buffer the entire request body into RAM
Tengine with XQUIC will buffer the entire response body into RAM
Jan 16, 2024
All read and write events of Tengine operate in Edge-Triggered mode. It is efficient and safe.
The directive proxy_buffering is enabled by default. You can disable the proxy_buffering specifically. Then, the response is passed to a client synchronously.
BTW, if the upstream response with the header "X-Accel-Buffering: no" specifically, even if the Tengine directive "proxy_buffering on" by default, the response will be passed to the client synchronously.
Thanks for the response. In my test case, I was serving a static file from the disk directly with Tengine (using the root directive). I was not using the reverse proxy functionality, so I do not believe that directive will fix the problem. I will try setting proxy_buffering in the test environment.
Additionally, the issue only occurs when using XQUIC. Using the same configuration, I can serve the same file with HTTP/2 and HTTP/1.1 without significant memory increase. I traced the buffering behavior with XQUIC to the function ngx_http_xquic_send_chain in gdb, and I do not believe it is possible to disable it with configuration.
Ⅰ. Issue Description
Tengine buffers the entire response body into RAM before sending it.
Ⅱ. Describe what happened
If serving a file, Tengine will read the entire file into RAM (visible from
htop
), and after that is done, it will send the file to the client.Ⅲ. Describe what you expected to happen
Tengine should not buffer the entire file in RAM. It should stream the file as it is read from the disk. Memory usage should not increase significantly.
Ⅳ. How to reproduce it (as minimally and precisely as possible)
ymuski/curl-http3
for local testing, but Firefox also triggers this)truncate -s 2G 2.bin
curl --http3 -v -k https://localhost/2.bin
. There is a substantial delay before curl begins to receive the response. It is visible fromhtop
that nginx will rapidly consume memory until it buffers the entire file in RAM.Ⅴ. Anything else we need to know?
It appears the issue is caused in part by
ngx_http_xquic_send_chain
reading the entire response into another chain. This prevents normal backpressure from happening, which results in the entire response being read into the buffer. The patch below makes it so that Tengine will no longer wait until the entire file is in RAM to begin sending the response. However, it does not fully fix the issue.With the patch, Tengine will still consume the same amount of RAM, but it will also busy-wait on
epoll_wait
. I am not sure how to fix this.Ⅵ. Environment:
sbin/nginx -V
): Commit 04baff4 on branchmaster
(latest commit as of issue creation)uname -a
):6.5.12-300.fc39.x86_64
The text was updated successfully, but these errors were encountered: