Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"HTTP error 502 while getting ..." #577

Closed
glyph opened this issue Aug 27, 2018 · 10 comments
Closed

"HTTP error 502 while getting ..." #577

glyph opened this issue Aug 27, 2018 · 10 comments

Comments

@glyph
Copy link

glyph commented Aug 27, 2018

I am using 4.5.0, but I see the behavior described on #449 every time I install a new version of a dependency for the first time.
@glyph
Copy link
Author

glyph commented Aug 27, 2018

For example, installing a recent setuptools update:

First attempt:

Collecting setuptools==40.2.0 (from -r requirements.txt (line 69))
  HTTP error 502 while getting http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl#sha256=ea3796a48a207b46ea36a9d26de4d0cc87c953a683a7b314ea65d666930ea8e6 (from http://127.0.0.1:3141/root/plus/+simple/setuptools/)
  Could not install requirement setuptools==40.2.0 from http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl#sha256=ea3796a48a207b46ea36a9d26de4d0cc87c953a683a7b314ea65d666930ea8e6 (from -r requirements.txt (line 69)) because of error 502 Server Error: Bad Gateway for url: http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl
Could not install requirement setuptools==40.2.0 from http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl#sha256=ea3796a48a207b46ea36a9d26de4d0cc87c953a683a7b314ea65d666930ea8e6 (from -r requirements.txt (line 69)) because of HTTP error 502 Server Error: Bad Gateway for url: http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl for URL http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl#sha256=ea3796a48a207b46ea36a9d26de4d0cc87c953a683a7b314ea65d666930ea8e6 (from http://127.0.0.1:3141/root/plus/+simple/setuptools/)

Second attempt, seconds later:

Collecting setuptools==40.2.0 (from -r requirements.txt (line 69))
  Downloading http://127.0.0.1:3141/root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl (568kB)
    100% |████████████████████████████████| 573kB 87.1MB/s

@fschulze
Copy link
Contributor

@glyph the ticket you mentioned is for releasing, not download.

Could you please run devpi-server with the --debug option and send us the log output around the time this error occurs? If you still have existing log output even without --debug it would be helpful.

Did this happen right after release on Aug 21st, or a bit later? I sometimes have issues right after making a new release of a package until the PyPI CDN has caught up.

You could also try the --replica-max-retries option (which should be renamed to --upstream-max-retries)

@glyph
Copy link
Author

glyph commented Aug 28, 2018

Hi @fschulze ! Thanks for the reply.

@glyph the ticket you mentioned is for releasing, not download.

Yeah, I was just referencing it in case there was some common 502 case; in retrospect, probably more confusing than useful.

Could you please run devpi-server with the --debug option and send us the log output around the time this error occurs? If you still have existing log output even without --debug it would be helpful.

Interesting. Searching through the logs for this version number, I see this on the 23rd:

2018-08-23 13:54:12,792 INFO  [req16134] GET /root/plus/+simple/setuptools/
2018-08-23 13:54:13,057 INFO  [req16134] [Wtx4534] setting projects cache for 'setuptools'
2018-08-23 13:54:13,074 INFO  [req16134] [Wtx4534] fswriter4535: committed: keys: 'root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl','root/pypi/setuptools/.simple','root/pypi/+f/5aa/f6f79b631f47f/setuptools-40.1.1.zip','root/pypi/+f/92e/126c856d02c54/setuptools-40.1.1-py2.py3-none-any.whl','root/pypi/+f/478/81d54ede4da9c/setuptools-40.2.0.zip'
2018-08-23 13:54:13,307 INFO  [req16135] GET /root/pypi/+f/d68/abee4eed409fb/setuptools-40.0.0-py2.py3-none-any.whl

and then this yesterday:

2018-08-27 15:31:37,524 INFO  [req19164] GET /root/plus/+simple/setuptools/
2018-08-27 15:31:37,822 INFO  [req19164] [Rtx4564] setting projects cache for 'setuptools'
2018-08-27 15:31:38,067 INFO  [req19165] GET /root/pypi/+f/ea3/796a48a207b46/setuptools-40.2.0-py2.py3-none-any.whl
2018-08-27 15:31:38,077 ERROR [req19165] [Wtx4564] Error during httpget of https://files.pythonhosted.org/packages/66/e8/570bb5ca88a8bcd2a1db9c6246bb66615750663ffaaeada95b04ffe74e12/setuptools-40.2.0-py2.py3-none-any.whl
Traceback (most recent call last):
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/devpi_server/main.py", line 287, in httpget
    timeout=timeout)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/Users/glyph/.local/venvs/devpi-server/lib/python3.6/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
2018-08-27 15:31:38,078 ERROR [req19165] [Wtx4564] error -1 getting https://files.pythonhosted.org/packages/66/e8/570bb5ca88a8bcd2a1db9c6246bb66615750663ffaaeada95b04ffe74e12/setuptools-40.2.0-py2.py3-none-any.whl

Did this happen right after release on Aug 21st, or a bit later? I sometimes have issues right after making a new release of a package until the PyPI CDN has caught up.

This was yesterday. The project I was installing had just bumped its dependency, but those log messages suggest to me that this was not the first time that DevPI had seen it!

You could also try the --replica-max-retries option (which should be renamed to --upstream-max-retries)

Ooh. This is exactly what I'm looking for.

The strange thing here is that pip directly doesn't seem to have this issue when talking to PyPI, but DevPI has it all the time. Perhaps pip simply has some internal retry logic that I'm not seeing? Anyway I'm going to add that option as a workaround and see if I have better luck in the future.

@mam8cc
Copy link

mam8cc commented Sep 26, 2018 •
edited

Hey @fschulze I'd like to hop in here and verify that we're experiencing this issue as well.

Server command:
devpi-server --replica-max-retries=3 --host 0.0.0.0 --port 4141 --serverdir devpi_files --debug

Stack trace of a 502 error the debug caught.

2018-09-26 16:20:05,385 INFO  [req3782] GET /es/prod/+simple/pyppeteer/
2018-09-26 16:20:05,429 INFO  [req3783] GET /root/pypi/+f/e84/8bf533f0d6cbc/Shapely-1.6.4.post1-cp36-cp36m-manylinux1_x86_64.whl
2018-09-26 16:20:05,456 ERROR [req3782] [Rtx11972] Error during httpget of https://pypi.python.org/simple/pyppeteer/
Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/devpi_server/main.py", line 287, in httpget
    timeout=timeout)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 640, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

We install pypeteer several times a day so it is neither a new install, nor is it consistently failing.

Versions we're running:

(venv) mike@devpi:/opt/devpi$ pip freeze | grep devpi
devpi-client==4.0.1
devpi-common==3.2.2
devpi-server==4.4.0
devpi-web==3.2.2

UPDATE:

I went ahead and updated us to latest devpi-server and we're still seeing the same 502 errors as I shared above:

(venv) mike@devpi:/opt/devpi$ pip freeze | grep devpi
devpi-client==4.0.1
devpi-common==3.3.1
devpi-server==4.5.0
devpi-web==3.2.2

and as you can see the pypi URL has been updated.

 2018-09-27 13:15:11,893 INFO  [req146851] GET /es/prod/+simple/infinity/
2018-09-27 13:15:11,896 ERROR [req146851] [Rtx456] Error during httpget of https://pypi.org/simple/infinity/
Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/devpi/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/devpi/venv/lib/python3.6/site-packages/devpi_server/main.py", line 287, in httpget
    timeout=timeout)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/devpi/venv/lib/python3.6/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Some additional information:

(venv) mike@devpi:~$ python --version
Python 3.6.3
(venv) mike@devpi:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

@fschulze
Copy link
Contributor

Please try with devpi-server 4.8.0

@glyph
Copy link
Author

glyph commented Nov 17, 2018

Hi @fschulze - I haven't seen this in a while, but I've long since added --replica-max-retries=10 to my devpi-server invocation. What behavior is different now exactly?

@fschulze
Copy link
Contributor

@glyph looking through your logs again the fix in 4.8.0 might not help. The fix is for big packages which was handled inefficiently in previous versions since streaming was added. Your issues seem to be with simple pages instead.

@fschulze
Copy link
Contributor

Seems like the issue is solved.

@joamatab
Copy link

Im using
devpi-server-5.0.0
devpi-web-3.5.1

and I experience the same issue that pip install package fails on the first attempt

 ERROR: HTTP error 502 while getting

is the fix invoking the devpi-server with --replica-max-retries=10?

@fschulze
Copy link
Contributor

@joamatab a 502 is a server side error, please try to find the traceback in the devpi-server logs and post it here with a few lines of context around it.

@eliba eliba mentioned this issue Nov 3, 2020
Closed
vermaport added a commit to vermaport/docker-devpi that referenced this issue May 4, 2021
@vermaport
Add --replica-max-retries 10
d53d74f 
1. Attempt to work around devpi/devpi#577
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fschulze @glyph @mam8cc @joamatab