Apache mod_proxy_uwsgi returning 502 when uwsgi returned 403 when using uwsgi over socket (EPIPE related?)

[johnsom]

Hello uwsgi community from the OpenStack community.

We have run into a problem that we haven't been able to track down, so I have decided to open a bug.

Summary: When running automated testing on an application using uwsgi behind Apache via the uwsgi protocol over a unix domain socket, we can regularly experience test failures where Apache returns a 502 to the user, but uwsgi (correctly) sent a 403. This only appears to happen on slower nested virtual machine environments.

Platform details:
Ubuntu Focal 20.04
Apache 2.4.41
uwsgi master branch e24fef1 (and older, but we reproduced it on this SHA as well)
With/without uwsgi built with debug enabled. Attached logs are from a debug build.
Using IPv4/IPv6 addressing for the client does not matter. Logs are from an IPv6 test run.
We tested with pull request 2208, it did not solve the problem.

Odd details we don't know if matter:
Seems to regularly happen on HTTP PATCH method calls.
Typically when calls are in rapid series.
When this occurs, we see rv == 104 (EPIPE) returned from this line of uwsgi_response() in mod_proxy_uwsgi.c:
https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_uwsgi.c?view=annotate#l390
rv = ap_get_brigade(rp->input_filters, bb, AP_MODE_READBYTES, mode, conf->io_buffer_size);

Logs and configuration files are in the attachment.

Interesting log lines:
screen-designate-api.txt (uwsgi log) -
Nov 13 00:21:27.039564
Nov 13 00:21:27.039564
--> Timestamp Nov 13 00:21:27.051166
2001:4801:7828:101:be76:4eff:fe10:1775 () {68 vars in 1569 bytes} [Sat Nov 13 00:21:27 2021] PATCH /dns/v2/quotas/ae3b0c5aaf0d4060b35e09d6c5f9d241 => generated 0 bytes in 4 msecs (HTTP/1.1 403) 4 headers in 170 bytes (1 switches on core 0)
We have a test that makes three calls in a row, all of which should return a 403. The third call is the one that ends up with a 502 back to the client.

access_log.txt (apache log)
Timestamp 13/Nov/2021:00:21:27 +0000
2001:4801:7828:101:be76:4eff:fe10:1775 - - [13/Nov/2021:00:21:27 +0000] "PATCH /dns/v2/quotas/ae3b0c5aaf0d4060b35e09d6c5f9d241 HTTP/1.1" 502 696 "-" "python-urllib3/1.26.7"

Things we have tried:
uwsgi: socket-timeout 30
apache ProxyPass with: disablereuse=On enablereuse=Off

Any ideas or insight would be appreciated.

uwsgi-502-logs-configs.tar.gz