gunicorn socket Ignoring EPIPE #1695

zhan9san · 2018-02-02T06:59:07Z

gunicorn == 0.17.4
Django == 1.5.1
nginx version: openresty/1.9.7.3

If I use "proxy_set_header Host $http_host;", django will raise SuspiciousOperation. So I add "proxy_set_header Host "${host}:${server_port}";" to nginx's config and set server_name naked ip.
It is based on this tips.django get_host

Why there is some difference when testing by curl and nc? Load balance uses the command nc to check whether gunicorn is ok. But its reponse code is not 2xx or 3xx when using "nc". 4xx is considered to be unhealthy.

Or is there some graceful configration to process the requests when using nc?

Thanks!

nginx-gunicorn.conf

upstream app_server {
server unix:/tmp/gunicorn.sock fail_timeout=0;
}
server {
    listen 8086;
    server_name 192.168.1.2;
    keepalive_timeout 5;
    proxy_ignore_client_abort on;
    access_log   logs/gunicorn.access.log;
    error_log     logs/gunicorn.error.log;
    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host "${host}:${server_port}";
        proxy_redirect off;
        proxy_pass http://app_server;
    }
}

gunicorn.conf.py

bind="unix:/tmp/gunicorn.sock"
workers = 1
pidfile = "/ebs/var/gunicorn/pidx.file"
loglevel = "debug"
accesslog = "/ebs/var/log/gunicornx_access.log"
errorlog = "/ebs/var/log/gunicornx_error.log"
pythonpath = "/ebs/zh/test_host1"
worker_connections = 200
max_requests = 1024
keepalive = 3
preload_app = True

supervisord.conf

[program:gunicorn]
command=/ebs/zh/bin/gunicorn -c /ebs/zh/test_host1/gunicorn.conf.py wsgi:application
directory=/ebs/zh/test_host1/test_host1
numprocs = 1
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile = /ebs/var/log/workers/test.log
stdout_logfile_maxbytes = 10KB
stdout_logfile_backups = 5
stderr_logfile = /ebs/var/log/workers/test.log

Test using curl

curl -I 192.168.1.2:8086/index
HTTP/1.1 301 MOVED PERMANENTLY
Server: st/
Date: Thu, 01 Feb 2018 19:33:37 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Location: http://192.168.1.2:8086/index/

gunicorn_error.log(gunicorn)

2018-02-01 13:33:37 [8221] [DEBUG] HEAD /index

gunicorn.access.log(nginx)

192.168.1.3 - - [02/Feb/2018:03:33:37 +0800] "HEAD /index HTTP/1.1" 301 0 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.26.0 OpenSSL/0.9.8b zlib/1.2.3 libidn/1.18"

Test using nc

echo -e 'HEAD /index HTTP/1.0\r\n\r\n' | nc -w 192.168.1.2 8086

gunicorn_error.log(gunicorn)

2018-02-01 12:40:58 [16203] [DEBUG] HEAD /index
2018-02-01 12:40:58 [16203] [DEBUG] Ignoring EPIPE

gunicorn.access.log(nginx)

192.168.1.3 - - [02/Feb/2018:02:57:43 +0800] "HEAD /index HTTP/1.0" 499 0 "-" "-"

The text was updated successfully, but these errors were encountered:

budimm · 2019-03-18T04:37:23Z

same with me,,
[DEBUG] Ignoring EPIPE

tilgovi · 2019-03-18T18:38:13Z

499 is NGINX is a client disconnect, I believe. That's also the same reason Gunicorn will log "Ignoring EPIPE". It sounds as though nc is not reading the full response.

tilgovi · 2019-03-18T18:41:37Z

Your nc command seems malformed, too, unless it's a copy-paste error. -w should specify a timeout.

budimm · 2019-03-19T00:36:25Z

Idk, but for me, its just warning, iam still could using my command using 'subprocess'

enderphan94 · 2019-05-04T10:02:11Z

getting the same issue :(

benoitc · 2019-05-04T19:49:11Z

@endrrphan94 how do you get this "issue" ? It's unclear for me for me since nc in this example doesn't send any a proper request.

boddumanohar · 2019-10-29T12:00:31Z

I too see the same error, but gunicorn recovers from this after sometime. I see this for a few minutes and then after sometime gunicorn starts taking requests. here are my gunicorn logs.

2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] GET /health/
 - - [29/Oct/2019:11:07:19 +0000] "GET /api-auth/login/ HTTP/1.1" 200 3010 "-" "ELB-HealthChecker/2.0"
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Closing connection. 
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:19.924Z [2019-10-29 11:07:19 +0000] [1858] [DEBUG] Ignoring EPIPE
2019-10-29T11:07:24.925Z [2019-10-29 11:07:24 +0000] [1858] [DEBUG] GET /health/
 - - [29/Oct/2019:11:07:24 +0000] "GET /api-auth/login/ HTTP/1.1" 200 3010 "-" "ELB-HealthChecker/2.0"
2019-10-29T11:07:24.926Z [2019-10-29 11:07:24 +0000] [1858] [DEBUG] Closing connection. 
2019-10-29T11:07:32.928Z [2019-10-29 11:07:32 +0000] [1858] [DEBUG] GET /health/
 - - [29/Oct/2019:11:07:32 +0000] "GET /api-auth/login/ HTTP/1.1" 200 3010 "-" "ELB-HealthChecker/2.0"
2019-10-29T11:07:32.928Z [2019-10-29 11:07:32 +0000] [1858] [DEBUG] Closing connection. 
2019-10-29T11:07:39.930Z [2019-10-29 11:07:39 +0000] [1858] [DEBUG] GET /health/
 - - [29/Oct/2019:11:07:39 +0000] "GET /api-auth/login/ HTTP/1.1" 200 3010 "-" "ELB-HealthChecker/2.0"
2019-10-29T11:07:39.930Z [2019-10-29 11:07:39 +0000] [1858] [DEBUG] Closing connection. 
2019-10-29T11:07:47.932Z [2019-10-29 11:07:47 +0000] [1858] [DEBUG] GET /health/
 - - [29/Oct/2019:11:07:47 +0000] "GET /api-auth/login/ HTTP/1.1" 200 3010 "-" "ELB-HealthChecker/2.0"
2019-10-29T11:07:47.932Z [2019-10-29 11:07:47 +0000] [1858] [DEBUG] Closing connection.

any idea why this happens?

benoitc · 2019-10-30T06:54:23Z

if you are using a synchronous worker, you should make sure to tell to elb to close the connection on each check . It seems it’s trying to retry on the same connextion.or maybe one other request is doing it.

…

On 29 Oct 2019, at 13:00, Manohar Reddy ***@***.***> wrote: I too see the same error, but gunicorn recovers from this after sometime. I see this for a few minutes and then after sometime gunicorn starts taking requests. here are my gunicorn logs. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

zhan9san · 2019-10-31T02:32:47Z

This issue cannot be reproduce these days. It's weird.
@tilgovi Sorry, It's typo

echo -e 'HEAD /index HTTP/1.0\r\n\r\n' | nc -t 192.168.1.2 8086

boddumanohar · 2019-10-31T02:59:31Z

@benoitc I am using worker class as gevent.

Earlier to this we were using the worker class as gthread we had the same issue where gunicorn has started dropping connection because of which elb health checks are failing. so we switched to genvent but problem still persists.

is there any suggestion that you can give to fix this?

benoitc · 2019-11-22T20:30:37Z

are you sending a close header? @boddumanohar

boddumanohar · 2020-04-17T05:39:01Z

@benoitc This happens when the ELB health check closed the request before gunicorn replies. The ELB health check has a timeout and within that timeout, if it doesn't receive any response. Its going to close the request.

So I think this is not a bug on gunicorn side.

benoitc · 2020-04-17T07:22:44Z

this epipe message is expected hence why it’s displayed in debug mode only. What is the issue there exactly?

zhan9san · 2020-04-28T17:12:00Z

I tried many times and still cannot reproduce this issue.
If others can't reproduce it either, i think this ticket can be closed.

dikokob · 2021-03-24T05:36:22Z

Any solutions as yet?

boddumanohar · 2021-03-24T05:49:59Z

this error usually occurs when the client closes the connection. So I believe there isn't much that can be done from the gunicorn side.

zhan9san · 2021-03-24T08:00:28Z

Hi @dikokob
Maybe @boddumanohar is correct. It's client issue.

Besides, this issue can be reproduced through this repo Reproduce this issue.

zhan9san · 2021-03-24T08:13:18Z

Besides, if proxy_ignore_client_abort on; is added in the Nginx proxy configuration, both 499 error from Nginx and Ignoring EPIPE from Gunicorn, are gone.

dikokob · 2021-03-24T14:47:23Z

@zhan9san I did that and it worked see answer

benoitc · 2021-03-25T06:35:23Z

Also again, what is the issue there? Apart displaying this log in debug mode? I

Ignorig aborted requests on Nginx is doing the same thing but one level above anyway, so that will remove useless calls to gunicorn which is fine. Would be good to have a pr to add it in the doc :)

zhan9san · 2021-03-25T08:44:23Z

Thanks for your attention.
I'll create a PR for this aborted request issue.

Fixes #1695

* disable sse due to a bug in gunicorn benoitc/gunicorn#1695 * UI regressions in assessview

benoitc assigned boddumanohar Nov 22, 2019

benoitc added Feature/Worker Feedback Requested labels Nov 22, 2019

boddumanohar removed their assignment Mar 24, 2021

benoitc closed this as completed in 917ebcd May 10, 2023

benoitc added a commit that referenced this issue May 10, 2023

Merge pull request #2548 from zhan9san/1695-ignore-aborted-requests

c829cc8

Fixes #1695

tuz666 mentioned this issue Nov 14, 2023

Adjust Nginx and Gunicorn config lahmacunradio/arcsi#186

Merged

b3n4kh added a commit to taranis-ai/taranis-ai that referenced this issue Apr 9, 2024

fix various frontend regressions:

07f5b69

* disable sse due to a bug in gunicorn benoitc/gunicorn#1695 * UI regressions in assessview

b3n4kh mentioned this issue Apr 9, 2024

SSE Not working with gunicorn taranis-ai/taranis-ai#196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gunicorn socket Ignoring EPIPE #1695

gunicorn socket Ignoring EPIPE #1695

zhan9san commented Feb 2, 2018

budimm commented Mar 18, 2019

tilgovi commented Mar 18, 2019

tilgovi commented Mar 18, 2019

budimm commented Mar 19, 2019

enderphan94 commented May 4, 2019

benoitc commented May 4, 2019

boddumanohar commented Oct 29, 2019 •

edited

Loading

benoitc commented Oct 30, 2019 via email

zhan9san commented Oct 31, 2019

boddumanohar commented Oct 31, 2019

benoitc commented Nov 22, 2019

boddumanohar commented Apr 17, 2020 •

edited

Loading

benoitc commented Apr 17, 2020

zhan9san commented Apr 28, 2020

dikokob commented Mar 24, 2021

boddumanohar commented Mar 24, 2021

zhan9san commented Mar 24, 2021

zhan9san commented Mar 24, 2021

dikokob commented Mar 24, 2021 •

edited

Loading

benoitc commented Mar 25, 2021 •

edited

Loading

zhan9san commented Mar 25, 2021

gunicorn socket Ignoring EPIPE #1695

gunicorn socket Ignoring EPIPE #1695

Comments

zhan9san commented Feb 2, 2018

nginx-gunicorn.conf

gunicorn.conf.py

supervisord.conf

Test using curl

Test using nc

budimm commented Mar 18, 2019

tilgovi commented Mar 18, 2019

tilgovi commented Mar 18, 2019

budimm commented Mar 19, 2019

enderphan94 commented May 4, 2019

benoitc commented May 4, 2019

boddumanohar commented Oct 29, 2019 • edited Loading

benoitc commented Oct 30, 2019 via email

zhan9san commented Oct 31, 2019

boddumanohar commented Oct 31, 2019

benoitc commented Nov 22, 2019

boddumanohar commented Apr 17, 2020 • edited Loading

benoitc commented Apr 17, 2020

zhan9san commented Apr 28, 2020

dikokob commented Mar 24, 2021

boddumanohar commented Mar 24, 2021

zhan9san commented Mar 24, 2021

zhan9san commented Mar 24, 2021

dikokob commented Mar 24, 2021 • edited Loading

benoitc commented Mar 25, 2021 • edited Loading

zhan9san commented Mar 25, 2021

boddumanohar commented Oct 29, 2019 •

edited

Loading

boddumanohar commented Apr 17, 2020 •

edited

Loading

dikokob commented Mar 24, 2021 •

edited

Loading

benoitc commented Mar 25, 2021 •

edited

Loading