502 Bad Gateway with gunicorn threads #3507
Description
Describe the issue
- Since 5.9, gunicorn runs with multiple threads by default.
- This configuration triggers an nginx "502 Bad Gateway" almost every time gunicorn restarts a worker (
--max-requests 7000 --max-requests-jitter 1000) during an http request, under moderate workload. - This error does NOT occur with gunicorn
--threads=1.
I already tried
- I've read and searched the documentation.
- I've searched for similar filed issues in this repository.
Steps to reproduce the behavior
- Start a fresh docker compose Weblate instance.
- To trigger the error faster, set
--max-requests 20and--max-requests-jitter 7in weblate_container:/etc/supervisor/conf.d/web.conf. - Restart weblate_container.
- Verify the gunicorn process looks like :
/app/venv/bin/python3 /app/venv/bin/gunicorn weblate.wsgi:application --preload --timeout 3600 --graceful-timeout 3600 --max-requests 20 --max-requests-jitter 7 --workers=2 --threads=2 --access-logfile=- --error-logfile=- --forwarded-allow-ips=* --bind unix:///run/gunicorn/app/weblate/socket - Trigger the error from a terminal :
read -sp "API key: " api_key
read -p "http host: " http_host
i=0
while true
do
i=$(($i+1))
echo "i[$i]"
curl --fail-with-body -H "Authorization: Token $api_key" https://$http_host/api/languages/ab/ || break
done; echo "api error after [$i] requests."
- After 1x to 3x
--max-requests:
...
{"id":2,"code":"ab","name":"Abkhazian","plural":{"id":2,...
{"id":2,"code":"ab","name":"Abkhazian","plural":{"id":2,...
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
curl: (22) The requested URL returned error: 502
api error after [26] requests.
- Try again with
--threads=1(for example by settingWEB_WORKERS=1) :
...
{"id":2,"code":"ab","name":"Abkhazian","plural":{"id":2,...
{"id":2,"code":"ab","name":"Abkhazian","plural":{"id":2,...
^C
Ok after 500+ requests.
The current curl request "freezes" every time a gunicorn worker restarts, but finally succeeds anyway.
Off course the problem also occurs with default --max-requests 7000, but is more difficult to reproduce quickly due to request throttling.
Expected behavior
A little freeze but no error on gunicorn max-requests worker restart.
I think Commit aae2514 should be rolledback, and the following config advertised in the documentation for low memory environments :
WEBLATE_WORKERS=1
WEB_WORKERS=2 # assuming --workers=%(ENV_WEB_WORKERS)s --threads=1
CELERY_SINGLE_PROCESS=1
As a bonus towards a more aggressive memory management, --max-requests could also be significantly reduced.
Something like : --max-requests 200 --max-requests-jitter 50
This would benefit a low memory environment.
A more powerful host should not see any impact because it can afford more workers.
Screenshots
No response
Exception traceback
- With "--threads=2" -> 502 nginx error :
...
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:26 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:26 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:27 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:27 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:34:27,152: INFO/444] Autorestarting worker after current request.
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:27 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:34:27,214: INFO/444] Worker exiting (pid: 444)
==> nginx stderr | 2025/08/03 14:34:27 [error] 281#281: *1693 writev() failed (32: Broken pipe) while sending request to upstream, client: ::ffff:192.168.122.10, server: , request: "GET /api/languages/ab/ HTTP/1.1", upstream: "http://unix:/run/gunicorn/app/weblate/socket:/api/languages/ab/", host: "weblate.foo.com"
==> nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:34:27 +0000] "GET /api/languages/ab/ HTTP/1.1" 502 150 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:34:27,882: INFO/677] Booting worker with pid: 677
nginx stdout | 172.18.0.1 - - [03/Aug/2025:14:34:28 +0000] "HEAD / HTTP/1.0" 301 0 "-" "-"
nginx stdout | 172.18.0.1 - - [03/Aug/2025:14:34:30 +0000] "HEAD / HTTP/1.0" 301 0 "-" "-"
nginx stdout | 172.18.0.1 - - [03/Aug/2025:14:34:32 +0000] "HEAD / HTTP/1.0" 301 0 "-" "-"
...
- With "--threads=1" -> ok :
...
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | 172.18.0.1 - - [03/Aug/2025:14:15:16 +0000] "HEAD / HTTP/1.0" 301 0 "-" "-"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:15:16,889: INFO/405] Autorestarting worker after current request.
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:15:16,899: INFO/405] Worker exiting (pid: 405)
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:16 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:15:17,331: INFO/406] Autorestarting worker after current request.
gunicorn stderr | [2025-08-03 14:15:17,342: INFO/406] Worker exiting (pid: 406)
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:17 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
gunicorn stderr | [2025-08-03 14:15:18,015: INFO/419] Booting worker with pid: 419
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:18 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:18 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:18 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
nginx stdout | ::ffff:192.168.122.10 - - [03/Aug/2025:14:15:18 +0000] "GET /api/languages/ab/ HTTP/1.1" 200 359 "-" "curl/8.5.0"
...Additional context
- Weblate 5.12.2
- default all-in-one docker compose
- on an Ubuntu 24.04 64bit VM with 2 cores and 5GB ram