"Connection reset by peer" error. #630

kimtree · 2013-10-15T00:19:23Z

Firstable, This is our run command.
/usr/bin/python /usr/local/bin/gunicorn -w 4 -b 127.0.0.1:5000 ext:app --log-level=error --error-logfile=ext_error.log

When I checked log in the morning, I could find some error logs like below.

{ERROR} /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:219 Error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 93, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 137, in handle_request
resp.write(item)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 313, in write
util.write(self.sock, arg, self.chunked)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 298, in write
sock.sendall(data)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
2013-10-15 08:05:43 [27468] [CRITICAL] WORKER TIMEOUT (pid:31362)
2013-10-15 08:05:43 [27468] [CRITICAL] WORKER TIMEOUT (pid:31362)

How can I figure out this error?

kimtree · 2013-10-15T01:11:35Z

After I digging this out, I think this issue coming from client side which is 499 http status code on nginx.
But I don't still know why this error makes gunicorn worker restarted.

benoitc · 2013-11-06T06:57:19Z

not sure either. The error is correctly handled in the loop and should go on the next accept

Maybe an OP in your app blocking the worker? What was the version of gunicorn?

kimtree · 2013-11-07T00:27:40Z

@benoitc It was 18.0. I think the error from long-running connection which use connections over 1 mins. After I set nginx and gunicorn timeout high, I couldn't see this logs but my doubt is still here.. why worker dead.

Avyd · 2013-12-26T02:01:49Z

I have got the same errors:

[XXXX] [ERROR] Error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn-18.2-py2.7.egg/gunicorn/workers/sync.py", line 93, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python2.7/dist-packages/gunicorn-18.2-py2.7.egg/gunicorn/workers/sync.py", line 137, in handle_request
resp.write(item)
File "/usr/local/lib/python2.7/dist-packages/gunicorn-18.2-py2.7.egg/gunicorn/http/wsgi.py", line 313, in write
util.write(self.sock, arg, self.chunked)
File "/usr/local/lib/python2.7/dist-packages/gunicorn-18.2-py2.7.egg/gunicorn/util.py", line 298, in write
sock.sendall(data)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
[CRITICAL] WORKER TIMEOUT (pid:XXXX)
[INFO] Booting worker with pid: XXXX

I'm also trying to figure out what happened.

benoitc · 2013-12-26T13:31:56Z

Do you have any op running more thn 30 secs by default? If yes, then the timeout is expected.

Avyd · 2013-12-27T03:42:47Z

What do you mean by op? I have set timeout limit in my gunicorn config file, but not sure if that's the cause.

benoitc · 2013-12-27T08:55:55Z

I mean operation sorry. Maybe one part of your code is doing a long operation that prevent the worker to notify the gunicorn arbiter. At some points the arbiter will consider that the worker timed out and will kill it.

An API call, a loop or another IO operation in your code are the usual suspect.

FreedomKnight · 2014-01-12T17:45:33Z

I also get 104 error when I use Python3

2014-01-13 01:40:34 [4367] [INFO] Starting gunicorn 18.0
2014-01-13 01:40:34 [4367] [INFO] Listening at: http://0.0.0.0:8000 (4367)
2014-01-13 01:40:34 [4367] [INFO] Using worker: sync
2014-01-13 01:40:34 [4370] [INFO] Booting worker with pid: 4370
2014-01-13 01:41:01 [4370] [ERROR] Error processing request.
Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/gunicorn/workers/sync.py", line 92, in handle
    req = six.next(parser)
  File "/usr/lib/python3.3/site-packages/gunicorn/http/parser.py", line 39, in __next__
    self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
  File "/usr/lib/python3.3/site-packages/gunicorn/http/message.py", line 152, in __init__
    super(Request, self).__init__(cfg, unreader)
  File "/usr/lib/python3.3/site-packages/gunicorn/http/message.py", line 49, in __init__
    unused = self.parse(self.unreader)
  File "/usr/lib/python3.3/site-packages/gunicorn/http/message.py", line 164, in parse
    self.get_data(unreader, buf, stop=True)
  File "/usr/lib/python3.3/site-packages/gunicorn/http/message.py", line 155, in get_data
    data = unreader.read()
   File "/usr/lib/python3.3/site-packages/gunicorn/http/unreader.py", line 38, in read
    d = self.chunk()
  File "/usr/lib/python3.3/site-packages/gunicorn/http/unreader.py", line 65, in chunk
    return self.sock.recv(self.mxchunk)
ConnectionResetError: [Errno 104] Connection reset by peer

My wsgi file just like the tutorial

def app(environ, start_response):
      data = "Hello, World!\n"
      start_response("200 OK", [
          ("Content-Type", "text/plain"),
          ("Content-Length", str(len(data)))
      ])
      return iter([data])

i save this file as index.py

i use the command python3-gunicorn -b 0.0.0.0:8000 index:app to run it and got the error message.
it work fine if i use python 2.

benoitc · 2014-01-12T18:03:58Z

@FreedomKnight how do you get this error? Also what if you run more than one worker (which is expected)

FreedomKnight · 2014-01-12T18:55:30Z

i thought i use browser to connect it and got this error at first time.
because my browser show this message

The webpage at http://localhost:8000/ might be temporarily down or it may have moved permanently to a new web address.
Error code: ERR_CONTENT_LENGTH_MISMATCH

it seems no coused by using browser.
I found i got error 104 when i use namp to scan localhost port.

and i add -w 4 to run more than one work, i still get same result.

benoitc · 2014-01-12T19:13:59Z

i don't reproduce it myself.... On python 3 you should send a binary and it should raise an error. What's the version of gunicorn?

FreedomKnight · 2014-01-12T22:51:33Z

I forgot it, it work when i change to b"200 ok"
it's my gunicorn version
python3-gunicorn (version 18.0)

benoitc · 2014-01-14T08:12:00Z

@FreedomKnight you should always pass a binary as a body kn Python 3. I will check the relevant code but it should raise an exception if you give a string.

FreedomKnight · 2014-01-14T14:07:33Z

i'm sorry about my stupid mistake, and thank you for your reply.
gunicorn is awesome.
btw, i think it really need raise exception ^^

expntly · 2014-03-06T07:56:27Z

Also seeing something "similar" in my logs, errno=9 though. gunicorn v17.5, workers=1, worker_class=gevent. According to the timestamps, it doesn't seem to be related to a timeout. What gives?

22486 2014-03-05 11:23:37,937 INFO glogging.py:274 "176.214.243.29 - - [05/Mar/2014:11:23:37] "GET /administrator/index.php HTTP/1.0" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
22486 2014-03-05 11:23:37,937 ERROR glogging.py:219 Socket error processing request.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/gunicorn/workers/async.py", line 42, in handle
req = six.next(parser)
File "/usr/lib/python2.6/site-packages/gunicorn/http/parser.py", line 39, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/lib/python2.6/site-packages/gunicorn/http/message.py", line 152, in init
super(Request, self).init(cfg, unreader)
File "/usr/lib/python2.6/site-packages/gunicorn/http/message.py", line 49, in init
unused = self.parse(self.unreader)
File "/usr/lib/python2.6/site-packages/gunicorn/http/message.py", line 164, in parse
self.get_data(unreader, buf, stop=True)
File "/usr/lib/python2.6/site-packages/gunicorn/http/message.py", line 155, in get_data
data = unreader.read()
File "/usr/lib/python2.6/site-packages/gunicorn/http/unreader.py", line 38, in read
d = self.chunk()
File "/usr/lib/python2.6/site-packages/gunicorn/http/unreader.py", line 65, in chunk
return self.sock.recv(self.mxchunk)
File "/usr/lib64/python2.6/site-packages/gevent/socket.py", line 273, in _dummy
raise error(EBADF, 'Bad file descriptor')
error: [Errno 9] Bad file descriptor

tilgovi · 2014-03-07T01:57:41Z

@expntly it looks like we ignore that error and discard the socket in the sync worker, but not the async worker base class. So that report makes sense. Thanks. I think this is different from the original report, though.

tilgovi · 2014-03-07T02:00:08Z

@benoitc my guess with the report from @expntly is that we return to the keepalive loop in some case when we should not.

fix benoitc#630 Conflicts: gunicorn/workers/sync.py

I often encounter benoitc/gunicorn#630, and update gunicorn to 19.1.1 can solve this error.

benoitc closed this as completed in 0e4d9f0 Mar 9, 2014

alinefr pushed a commit to alinefr/gunicorn that referenced this issue Apr 24, 2014

make sure we return the correct error with the last traceback.

874bd94

fix benoitc#630 Conflicts: gunicorn/workers/sync.py

jdelic mentioned this issue Oct 24, 2014

"Connection reset by peer" in handle_request isn't handled (or ignored) #921

Closed

sbwdlihao added a commit to sbwdlihao/simplecoin_multi that referenced this issue Nov 13, 2014

fix "gunicorn error 104 Connection reset by peer"

d5fd463

I often encounter benoitc/gunicorn#630, and update gunicorn to 19.1.1 can solve this error.

sbwdlihao mentioned this issue Nov 13, 2014

fix "gunicorn error 104 Connection reset by peer" simplecrypto/simplecoin_multi#165

Merged

nickgzzjr pushed a commit to nickgzzjr/simplecoin_multi that referenced this issue Oct 5, 2015

fix "gunicorn error 104 Connection reset by peer"

0cfda60

I often encounter benoitc/gunicorn#630, and update gunicorn to 19.1.1 can solve this error.

juntaozhang mentioned this issue Aug 7, 2017

fix "gunicorn error 104 Connection reset by peer" open-falcon/dashboard#59

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Connection reset by peer" error. #630

"Connection reset by peer" error. #630

kimtree commented Oct 15, 2013

kimtree commented Oct 15, 2013

benoitc commented Nov 6, 2013

kimtree commented Nov 7, 2013

Avyd commented Dec 26, 2013

benoitc commented Dec 26, 2013

Avyd commented Dec 27, 2013

benoitc commented Dec 27, 2013

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 12, 2014

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 12, 2014

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 14, 2014

FreedomKnight commented Jan 14, 2014

expntly commented Mar 6, 2014

tilgovi commented Mar 7, 2014

tilgovi commented Mar 7, 2014

"Connection reset by peer" error. #630

"Connection reset by peer" error. #630

Comments

kimtree commented Oct 15, 2013

kimtree commented Oct 15, 2013

benoitc commented Nov 6, 2013

kimtree commented Nov 7, 2013

Avyd commented Dec 26, 2013

benoitc commented Dec 26, 2013

Avyd commented Dec 27, 2013

benoitc commented Dec 27, 2013

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 12, 2014

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 12, 2014

FreedomKnight commented Jan 12, 2014

benoitc commented Jan 14, 2014

FreedomKnight commented Jan 14, 2014

expntly commented Mar 6, 2014

tilgovi commented Mar 7, 2014

tilgovi commented Mar 7, 2014