Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colima still crashing under network load #603

Closed
1 of 5 tasks
github-nicolas-stadler opened this issue Feb 1, 2023 · 6 comments
Closed
1 of 5 tasks

Colima still crashing under network load #603

github-nicolas-stadler opened this issue Feb 1, 2023 · 6 comments
Milestone

Comments

@github-nicolas-stadler
Copy link

Description

My lima VM crashes as soon as network activity gets a little high (such as a Sabnzbd download starting).

Version

Colima Version: 0.5.0 (also tried with HEAD)
Lima Version: 0.14.2
Qemu Version:

Operating System

  • macOS Intel <= 12 (Monterrey)
  • macOS Intel >= 13 (Ventura)
  • macOS M1 <= 12 (Monterrey)
  • macOS M1 >= 13 (Ventura)
  • Linux

Output of colima status

No output. The command hangs forever unless I run killall limactl

Reproduction Steps

  1. Start colima with the command provided
  2. Start a couple of downloads via Sabnzbd/Torrent
  3. VM hangs and docker stops responding until I manually kill limactl

Expected behaviour

VM should be able to sustain network activity under some load.

Additional context

My Colima command:

$HOMEBREW_BIN_PATH/colima start --activate \
  --arch aarch64 \
  --cpu 8 \
  --disk 494 \
  --dns 1.1.1.1 \
  --dns 1.0.0.1 \
  --memory 16 \
  --mount $DOWNLOADS_PATH:w \
  --mount $SERVER_PATH:w \
  --mount $SERVER_DATA_PATH:w \
  --mount-type virtiofs \
  --network-driver slirp \
  --runtime docker \
  --ssh-agent \
  --vm-type vz

I tried slirp and gvproxy for the network driver but it doesn't seem to make much of a difference.
Interesting to note, Docker Desktop for Mac had similar issues but they seem to have been fixed in version 4.16 (no more crashes for me since then): docker/for-mac#6530

@abiosoft
Copy link
Owner

abiosoft commented Feb 1, 2023

I believe this is specific to the vz driver and related to this issue lima-vm/lima#1200.

I will try to set out some time to troubleshoot the network and probably find an alternative network for the vz driver.

@Curtisjk
Copy link

I am facing the same issue when pulling large docker containers.

@AndreasA
Copy link

just had the same issue. doing a docker pull / composer install after using scp to retrieve files on a docker volume inside a container. Afterwards colima was nearly useless as starting the instace took forever due to #690 and also composer install never seemed to finish anymore afterwards.

but as mentioned by @abiosoft it is probably a lima and not a colima issue.

@abiosoft
Copy link
Owner

@AndreasA are you using VZ or QEMU?

@AndreasA
Copy link

@abiosoft I am using VZ. so I guess it is probably the lima issue?

@mig8447
Copy link

mig8447 commented Apr 25, 2023

I'm having the exact same issue when using a qemu VM, it doesn't happen with all downloads, but the terminal hangs and I need to kill the ssh process related to the VM and restart colima

Note that this happens only when running inside of a container, building seems to work fine and downloads could be anything from a package to a single file. I haven't been able to determine the exact cause of this. I attributed it to bind mounts and unpacking a file containing too many files (Using docker volumes seems to work fine) but I don't know anymore

@abiosoft abiosoft added this to the v0.6.0 milestone Nov 12, 2023
jesse-c pushed a commit to SeldonIO/MLServer that referenced this issue May 30, 2024
* build: Lock GitHub runners' OS

This was motivated by our macOS jobs failing [2] because
colima is missing. It looks like this is because the
latest versions of the macOS runner no longer have
colima installed by default [1].

colima is now explicitly installed.

[1] actions/runner-images#6216
[2] `/Users/runner/work/_temp/f19ffbff-27a9-4fc7-80b6-97791d2de141.sh: line 9: colima: command not found`

* build: Lock Colima

* build: Move macOS Docker installation to script

* build: Move macOS libomp activation to script

* build: Use latest Colima

The > 0.6.0 releases actually fix the issue we have linked [1][2][3].

[1] abiosoft/colima#577
[2] https://github.com/jesse-c/MLServer/blob/c3acd60995a72141027eff506e4fd330fe824179/hack/install-docker-macos.sh#L18-L20
[3] > Switch to new user-v2 network. Fixes abiosoft/colima#648, abiosoft/colima#603, abiosoft/colima#577, abiosoft/colima#779, abiosoft/colima#137, abiosoft/colima#740.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants