Tengine with XQUIC will buffer the entire response body into RAM

[iczero]

Ⅰ. Issue Description

Tengine buffers the entire response body into RAM before sending it.

Ⅱ. Describe what happened

If serving a file, Tengine will read the entire file into RAM (visible from htop), and after that is done, it will send the file to the client.

Ⅲ. Describe what you expected to happen

Tengine should not buffer the entire file in RAM. It should stream the file as it is read from the disk. Memory usage should not increase significantly.

Ⅳ. How to reproduce it (as minimally and precisely as possible)

1. Compile Tengine with XQUIC support
2. Acquire QUIC capable HTTP client (I used docker image ymuski/curl-http3 for local testing, but Firefox also triggers this)
3. Configure Tengine to serve files from a directory
4. Create a large file, or truncate -s 2G 2.bin
5. curl --http3 -v -k https://localhost/2.bin. There is a substantial delay before curl begins to receive the response. It is visible from htop that nginx will rapidly consume memory until it buffers the entire file in RAM.

Ⅴ. Anything else we need to know?

It appears the issue is caused in part by ngx_http_xquic_send_chain reading the entire response into another chain. This prevents normal backpressure from happening, which results in the entire response being read into the buffer. The patch below makes it so that Tengine will no longer wait until the entire file is in RAM to begin sending the response. However, it does not fully fix the issue.

diff --git a/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c b/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c
index 6eff9254..8b0fb707 100644
--- a/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c
+++ b/modules/ngx_http_xquic_module/ngx_http_xquic_filter_module.c
@@ -648,6 +648,9 @@ ngx_http_xquic_send_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
     /* update h3_stream->output_queue here */
     h3_stream = r->xqstream;
 
+    // HACK: stop accepting new buffers if we have been limited by flow control
+    if (!h3_stream->wait_to_write) {
+
     last_chain = h3_stream->output_queue;
 
     while (last_chain != NULL) {
@@ -705,10 +708,12 @@ ngx_http_xquic_send_chain(ngx_connection_t *c, ngx_chain_t *in, off_t limit)
         r->xqstream->queued++; /* used to count buffers not sent*/
     }
 
+    // end hack
+    }
 
     last_out = h3_stream->output_queue;
     send = 0;

With the patch, Tengine will still consume the same amount of RAM, but it will also busy-wait on epoll_wait. I am not sure how to fix this.

Ⅵ. Environment:

• Tengine version (use sbin/nginx -V): Commit 04baff4 on branch master (latest commit as of issue creation)
• OS (e.g. from /etc/os-release): Fedora 39
• Kernel (e.g. uname -a): 6.5.12-300.fc39.x86_64
• Others: XQUIC commit alibaba/xquic@4d024a6

[lianglli]

All read and write events of Tengine operate in Edge-Triggered mode. It is efficient and safe.
The directive proxy_buffering is enabled by default. You can disable the proxy_buffering specifically. Then, the response is passed to a client synchronously.

proxy_buffering                         off;
proxy_buffer_size                     64k;
proxy_buffers                           256 64k;

BTW, if the upstream response with the header "X-Accel-Buffering: no" specifically, even if the Tengine directive "proxy_buffering on" by default, the response will be passed to the client synchronously.

[iczero]

Thanks for the response. In my test case, I was serving a static file from the disk directly with Tengine (using the root directive). I was not using the reverse proxy functionality, so I do not believe that directive will fix the problem. I will try setting proxy_buffering in the test environment.

Additionally, the issue only occurs when using XQUIC. Using the same configuration, I can serve the same file with HTTP/2 and HTTP/1.1 without significant memory increase. I traced the buffering behavior with XQUIC to the function ngx_http_xquic_send_chain in gdb, and I do not believe it is possible to disable it with configuration.

I can provide the full configuration if needed.