Impossible to not Buffer Request Body with NGINX #2121
Fill in as much as possible so that we can understand, find and fix the problem.
Are you sure this is a bug in Passenger? Yes. Passenger does not expose a way to configure this flag for the nginx module.
Question 1: What is the problem?
When attempting to process a request in a streaming fashion NGINX is currently buffering the entire request, before making a connection to the downstream app. This is controlled by the setting:
There's a couple NGINX modules that then set it back to 1 to support streaming. For example HERE in the proxy module.
However passenger doesn't seem to reference this setting anywhere, even when just grepping the repo:
This leads to the request always being buffered in nginx. Since this functionally defaults to 0 in nginx. This should happen with any architecture, but we've provided a reproduction with a simple nodejs app.
Passenger with nginx should expose a setting to allow nginx to not buffer requests before sending it on.
Passenger does not expose this setting.
Be as detailed as possible in your descriptions, include any logs and stack traces (don't just cut/paste the error, provide some logging before that too).
(if you are requesting a feature instead of reporting an issue, describe here what you have in mind and how it would help you)
HERE is a gist for the nginx configuration/single node app file.
All I do is startup the docker file by building it, and then running it:
docker build . -t passenger-test && docker run --rm -p 8080:80 -i -t passenger-test
Then in another terminal I generate an empty large file (easy to see with files 15GB or more), and try to upload it:
dd if=/dev/zero of=output.dat bs=1M count=15000 curl -H "Transfer-Encoding: chunked" -d @output.dat -XPOST http://localhost:8080
You'll notice that even though curl is actively doing work. The server won't log the connection exists for quite awhile. This is because passenger is buffering the full file before passing it on to the node application.
Question 2: Passenger version and integration mode
Question 3: OS or Linux distro, platform (including version):
Question 4: Passenger installation method:
Question 5: Your app's programming language (including any version managers) and framework (including versions):
Question 6: Are you using a PaaS and/or containerization? If so which one?
Question 7: Anything else about your setup that we should know?
Your answer: Not that I can think of.
The text was updated successfully, but these errors were encountered:
Work has begun on this issue.
Initial investigation reveals that implementing this feature requires not only setting the
We now always set request_body_no_buffering. We rely on the Core Controller's own buffering instead. The biggest issue I had to deal with so far was writing an output filter. This output filter is installed when dealing with chunked request bodies, and re-chunks data sent to the Controller (because Nginx dechunks bodies). The following integration tests still fail: Phusion Passenger for Nginx a Ruby app running on the root URI it should behave like an example web app buffers uploads [ci:skip]
The investigation today revealed that part of the reason why my change from last time did not work, is due to a bug in Nginx itself. Specifically, nginx/nginx@62821f1, which is part of version 1.15.3. The bug manifests itself when more the request body is received over more than one request buffer. The bug in Nginx causes Nginx to enter an infinite loop due to the fact that it incorrectly iterates over a linked list.
This means that if we implement this feature request, then we need to bump our minimum supported Nginx version to 1.15.3.