Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection reset by peer when running pip in a Colima powered Docker Container #577

Closed
1 of 5 tasks
Ponsaille opened this issue Jan 10, 2023 · 7 comments
Closed
1 of 5 tasks
Milestone

Comments

@Ponsaille
Copy link

Ponsaille commented Jan 10, 2023

Description

Most of the time colima works perfectly and thank you for that.

However I keep coming across an error when I use pip (python's package manager) to install dependencies. After a time it outputs this error or something similar:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 437, in _error_catcher
    yield
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
    data = self._fp_read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 526, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
    data = self.__fp.read(amt)
  File "/usr/local/lib/python3.8/http/client.py", line 459, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.8/http/client.py", line 503, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 400, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
    super().__init__(
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
    self.dist = self._prepare()
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
    dist = self._prepare_distribution()
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url
    file = get_http_url(
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/download.py", line 147, in __call__
    for chunk in chunks:
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
    for chunk in iterable:
  File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 621, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 586, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

This happens only when using it from colima. I have tried on Ubuntu and this issue has never appeared.

I have tried a large number of fixes that where recommended online including setting the DNS of colima to 8.8.8.8 or installing the python package pyopenssl. Installing the packages directly in the alpine vm also seam to work.

I have not been able to find any differentiating factor so this is why I am opening the issue here. I hope you will be able to help me.

Version

Colima Version:
colima version 0.5.2
git commit: 6b5b6fe

runtime: docker
arch: x86_64
client: v20.10.19
server: v20.10.20

Lima Version: 0.14.2
Qemu Version: 7.2.0

Operating System

  • macOS Intel <= 12 (Monterrey)
  • macOS Intel >= 13 (Ventura)
  • macOS M1 <= 12 (Monterrey)
  • macOS M1 >= 13 (Ventura)
  • Linux

Output of colima status

INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/ponsaijj/.colima/default/docker.sock

Reproduction Steps

Example of requirements.txt on which it has failed:

absl-py==0.13.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
affine==2.3.0; python_version >= "3.6"
aiobotocore==1.3.3; python_version >= "3.6"
aiohttp==3.7.4.post0; python_version >= "3.6"
aioitertools==0.8.0; python_version >= "3.6"
async-timeout==3.0.1; python_version >= "3.6" and python_full_version >= "3.5.3"
attrs==21.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
boto3==1.17.106; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.6.0"
botocore==1.20.106; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
cachetools==4.2.2; python_version >= "3.5" and python_version < "4.0" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
certifi==2021.5.30; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
chardet==4.0.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.5.0"
charset-normalizer==2.0.3; python_full_version >= "3.6.0" and python_version >= "3"
clearml==1.0.4
click-plugins==1.1.1; python_version >= "3.6"
click==7.1.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6" and python_version < "4"
cligj==0.7.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version < "4" and python_version >= "3.6"
colorama==0.4.4; python_version >= "3.6" and python_full_version < "3.0.0" and platform_system == "Windows" or python_full_version >= "3.5.0" and python_version >= "3.6" and platform_system == "Windows"
cycler==0.10.0; python_version >= "3.7"
efficientnet-pytorch==0.6.3; python_full_version >= "3.5.0"
fsspec==2021.7.0; python_version >= "3.6"
furl==2.1.2
future==0.18.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
google-auth-oauthlib==0.4.4; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
google-auth==1.33.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
grpcio==1.39.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
humanfriendly==9.2; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
idna==3.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
imagecorruptions==1.1.2
imageio==2.9.0; python_version >= "3.7"
imgaug==0.4.0
importlib-metadata==4.6.1; python_version >= "3.6" and python_version < "3.8" and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.2.0" and python_version >= "3.6" and python_version < "3.8")
jmespath==0.10.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.6.0"
jsonschema==3.2.0
kiwisolver==1.3.1; python_version >= "3.7"
markdown==3.3.4; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
matplotlib==3.4.2; python_version >= "3.7"
multidict==5.1.0; python_version >= "3.6"
munch==2.5.0; python_full_version >= "3.0.0"
networkx==2.6.1; python_version >= "3.7"
numpy==1.21.1; python_version >= "3.7" and python_full_version >= "3.6.1" and (python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.7")
oauthlib==3.1.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
opencv-python==4.5.3.56; python_version >= "3.6"
orderedmultidict==1.0.1
packaging==21.0; python_version >= "3.6"
pandas==1.1.5; python_full_version >= "3.6.1" and python_version >= "3.7"
pathlib2==2.3.6
pillow==8.3.1; python_version >= "3.7" and python_full_version >= "3.0.0"
pretrainedmodels==0.7.4; python_full_version >= "3.0.0"
protobuf==3.17.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
psutil==5.8.0; python_version >= "2.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
pyasn1-modules==0.2.8; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
pyasn1==0.4.8; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "4" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") or python_full_version >= "3.6.0" and python_version >= "3.6" and python_version < "4" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
pydeprecate==0.3.0; python_version >= "3.6"
pyjwt==2.1.0; python_version >= "3.6"
pyparsing==2.4.7; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.7"
pyreadline==2.1; python_version >= "2.7" and python_full_version < "3.0.0" and sys_platform == "win32" or python_full_version >= "3.5.0" and sys_platform == "win32"
pyrsistent==0.18.0; python_version >= "3.6"
python-dateutil==2.8.2; python_full_version >= "3.6.1" and python_version >= "3.7" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
python-dotenv==0.19.0; python_version >= "3.5"
pytorch-lightning==1.3.8; python_version >= "3.6"
pytz==2021.1; python_full_version >= "3.6.1" and python_version >= "3.7"
pywavelets==1.1.1; python_version >= "3.7"
pyyaml==5.4.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
rasterio==1.2.6; python_version >= "3.6"
requests-oauthlib==1.3.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
requests==2.26.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.6.0"
rsa==4.7.2; python_version >= "3.5" and python_version < "4" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
s3fs==2021.7.0; python_version >= "3.6"
s3transfer==0.4.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.6.0"
scikit-image==0.18.2; python_version >= "3.7"
scipy==1.6.0; python_version >= "3.7"
segmentation-models-pytorch==0.2.0; python_full_version >= "3.0.0"
shapely==1.7.1
six==1.16.0; python_full_version >= "3.3.0" and python_version >= "3.6" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and (python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.7") and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0")
snuggs==1.4.7; python_version >= "3.6"
tensorboard-plugin-wit==1.8.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
tensorboard==2.4.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
tifffile==2021.7.2; python_version >= "3.7"
timm==0.4.12; python_version >= "3.6" and python_full_version >= "3.0.0"
torch==1.9.0; python_full_version >= "3.6.2"
torchmetrics==0.4.1; python_version >= "3.6"
torchvision==0.10.0; python_version >= "3.6" and python_full_version >= "3.0.0"
tqdm==4.61.2; python_full_version >= "3.4.0" and python_version >= "3.6"
typer==0.3.2; python_version >= "3.6"
typing-extensions==3.10.0.0; python_full_version >= "3.6.2" and python_version >= "3.6" and python_version < "3.8"
urllib3==1.26.6; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.6"
werkzeug==2.0.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.2.0" and python_version >= "3.6"
wrapt==1.12.1; python_version >= "3.6"
yarl==1.6.3; python_version >= "3.6"
zipp==3.5.0; python_version >= "3.6" and python_version < "3.8"
pytest-cov==2.12.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
pytest-mock==3.6.1; python_version >= "3.6"
pytest==6.2.4; python_version >= "3.6"

Dockerfile:

FROM prefecthq/prefect:1.2.2-python3.8

COPY requirements.txt /tmp
WORKDIR /tmp

RUN pip install -U pip
RUN pip install -r requirements.txt

Expected behaviour

Packages installation should go through the end.

Additional context

No response

@abiosoft
Copy link
Owner

@Ponsaille does the issue go away after a Colima restart?

Also, can you can try --network-driver slirp and see if the problem persists?

@Ponsaille
Copy link
Author

@abiosoft using --network-driver slirp is working perfectly. I think we can close the issue. Thank you!

Out of curiosity, would you have any resource for me to understand how using slirp made this work?

@abiosoft
Copy link
Owner

For QEMU, Colima has two options for the network.

Slirp is the default QEMU user-mode network while GVProxy is an alternative.

It used to be only slirp in Colima but complains from users led to the trial of gvproxy and it turned out positive.

Neither has been perfect so far but judging from issues reported by users, gvproxy seems to work for most people and is thereby made the default.

@dacort
Copy link

dacort commented Feb 15, 2023

Was also running into this on macOS Intel 13.2 - network dropping both in VM on host and eventually coming back. The --network-driver slirp option seems to help for now! still exhibits the issue.

➜ limactl --version
limactl version 0.14.2

➜ colima version
colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

runtime: docker
arch: x86_64
client: v20.10.22
server: v20.10.20

@mattiapiccinetti
Copy link

@abiosoft using --network-driver slirp also worked for me, same issue but on macOS M1 >= 13 (Ventura). Thanks.

@marino-serna
Copy link

Was also running into this on macOS Intel 13.2 - network dropping both in VM on host and eventually coming back. The --network-driver slirp option seems to help for now! still exhibits the issue.

➜ limactl --version
limactl version 0.14.2

➜ colima version
colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

runtime: docker
arch: x86_64
client: v20.10.22
server: v20.10.20

Same here, if the mac uses intel, it works perfect with macOS 11 (Big Sur) but produce this error both with macOS 12 and 13 (Monterey and Ventura)

Changing the network-driver do not solve the issue.

Have you Dacort find a solution?

@dacort
Copy link

dacort commented Aug 4, 2023

@marino-serna No, I haven't. :\ I actually just had my entire VM freeze when I was trying to pull a new docker image. Things had actually been working pretty for me for a while, so not sure what happened. Here's my versions/status.

❯ limactl --version
limactl version 0.15.1

❯ colima version
colima version 0.5.4
git commit: feef4176f56a7dea487d43689317a9d7fe9de27e

runtime: docker
arch: x86_64
client: v23.0.1
server: v20.10.20

❯ colima status
INFO[0000] colima is running using macOS Virtualization.Framework 
INFO[0000] arch: x86_64                                 
INFO[0000] runtime: docker                              
INFO[0000] mountType: virtiofs                          
INFO[0000] socket: unix:///Users/dacort/.colima/default/docker.sock 

@abiosoft abiosoft added this to the v0.6.0 milestone Nov 12, 2023
jesse-c pushed a commit to SeldonIO/MLServer that referenced this issue May 30, 2024
* build: Lock GitHub runners' OS

This was motivated by our macOS jobs failing [2] because
colima is missing. It looks like this is because the
latest versions of the macOS runner no longer have
colima installed by default [1].

colima is now explicitly installed.

[1] actions/runner-images#6216
[2] `/Users/runner/work/_temp/f19ffbff-27a9-4fc7-80b6-97791d2de141.sh: line 9: colima: command not found`

* build: Lock Colima

* build: Move macOS Docker installation to script

* build: Move macOS libomp activation to script

* build: Use latest Colima

The > 0.6.0 releases actually fix the issue we have linked [1][2][3].

[1] abiosoft/colima#577
[2] https://github.com/jesse-c/MLServer/blob/c3acd60995a72141027eff506e4fd330fe824179/hack/install-docker-macos.sh#L18-L20
[3] > Switch to new user-v2 network. Fixes abiosoft/colima#648, abiosoft/colima#603, abiosoft/colima#577, abiosoft/colima#779, abiosoft/colima#137, abiosoft/colima#740.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants