Containers on Apple Silicon with emulated x86 are not reachable from outside

[Zordid]

Description

I am having a problem that is described in many other issues here - none with a proper solution. Some that gave me hope I could work around this, but sadly nothing works so far.

I am simply emulating x86 on my Apple M3.

Starting colima like this:
colima start default --arch x86_64 --cpu 4 --memory 4 --disk 20 --vm-type=vz-rosetta

Works fine, but now I already learned that this might be a problem:

❯ colima list
PROFILE    STATUS     ARCH      CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    x86_64    4       4GiB      20GiB    docker

There is no address exposed.

After spinning up some docker-compose containers, the situation that gives the problem is basically that no ports are reachable:

❯ docker port 376c82ad5d8d
9092/tcp -> 0.0.0.0:9092
9092/tcp -> [::]:9092
❯ nc -zv localhost 9092
nc: connectx to localhost port 9092 (tcp) failed: Connection refused
nc: connectx to localhost port 9092 (tcp) failed: Connection refused

I have now followed suggestions to disable v6, which did not help at all, I have played around with the flag --network-address which did not help at all.

The only valid next option I got now is to downgrade colima to an older version which also was one of the suggested solutions. But this I would constitute would mean there's a bug in Colima 0.6.8, right?

Version

0.6.8

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

INFO[0000] colima [profile=default] is running using macOS Virtualization.Framework
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/Q163557/.colima/x86_64/docker.sock

Reproduction Steps

1. start colima colima start default --arch x86_64 --cpu 4 --memory 4 --disk 20 --vm-type=vz-rosetta
2. see that no address is exposed colima list
3. run any docker container with exposed ports - try to reach them.

Expected behaviour

you should be able to reach all exposed ports that docker shows when docker port shows it

Additional context

No response

[Zordid]

I now downgraded to Colima 0.5.6 - the only option that is obviously missing is --vm-type=vz-rosetta - but after this downgrade, everything is just working fine. :( So there must be an issue in the 0.6.x branch of some sort.

[Zordid]

Here are some other related issues:
#962
#801
#897

All of those have the same or at least a seemingly identical problem. Some said, they fixed it disabling ipv6, which did nothing for me, others suggest restarting the Mac, which I did not try because I wanted to find the root cause and downgraded to an older version of Colima instead, which was successful on the first try.

[oguzhanakan0]

Thanks for creating this issue. I've been facing this issue on M3 chip Mac and after hours of start/stop cycle I downgraded colima as well.

[JackieTreeh0rn]

same issue here with emulated x86_64 on M1 Mac. disabling IPv6 did not help.

[JackieTreeh0rn]

I can confirm that using --profile instead of --arch works, the address is listed as well under colima list

ie:
colima start --profile x86_64 --vm-type=vz --vz-rosetta --cpu 8 --memory 8 --disk 10 --network-address

[Zordid]

I can confirm that using --profile instead of --arch works, the address is listed as well under colima list

ie: colima start --profile x86_64 --vm-type=vz --vz-rosetta --cpu 8 --memory 8 --disk 10 --network-address

This is not possible, simply because --profile is just giving the thing a different name, it has nothing to do with --arch. I don't know why you think this is the solution, but no, it isn't. Naming something "x86_64" does not make it emulate this architecture at all. Maybe that's why you think it is working?

[kento1218]

Just updated from v0.6.7 to v0.6.8 and I'm facing the same problem.
It's weired but skipping --vm-type=vz option works for me (Ofcourse I prefer to use it though)

% colima start amd64 --arch x86_64                          
INFO[0000] starting colima [profile=amd64]              
INFO[0000] runtime: docker                              
INFO[0000] creating and starting ...                     context=vm
INFO[0075] provisioning ...                              context=docker
INFO[0076] starting ...                                  context=docker
INFO[0147] done                                         

% colima start amd64-r --arch x86_64 --vm-type=vz --vz-rosetta
INFO[0000] starting colima [profile=amd64-r]            
INFO[0000] runtime: docker                              
INFO[0000] creating and starting ...                     context=vm
> [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/kento/.colima/_lima/colima-amd64-r/ga.sock" (host)
> [hostagent] connection to the guest agent was closed unexpectedlydial unix /Users/kento/.colima/_lima/colima-amd64-r/ga.sock: connect: no such file or directory
> [hostagent] The final requirement 1 of 1 is satisfied
> [guest agent does not seem to be running; port forwards will not work]
> DEGRADED. The VM seems running, but file sharing and port forwarding may not work. (hint: see "/Users/kento/.colima/_lima/colima-amd64-r/ha.stderr.log")
> degraded, status={Running:true Degraded:true Exiting:false Errors:[guest agent does not seem to be running; port forwards will not work] SSHLocalPort:50227}
FATA[0124] error starting vm: error at 'creating and starting': exit status 1 

@ M2, Ventura 13

[JackieTreeh0rn]

I can confirm that using --profile instead of --arch works, the address is listed as well under colima list
ie: colima start --profile x86_64 --vm-type=vz --vz-rosetta --cpu 8 --memory 8 --disk 10 --network-address

This is not possible, simply because --profile is just giving the thing a different name, it has nothing to do with --arch. I don't know why you think this is the solution, but no, it isn't. Naming something "x86_64" does not make it emulate this architecture at all. Maybe that's why you think it is working?

I just realizes this, yes. The address was exposed but not emulating.

[WandersonAlves]

I'm on colima 0.6.7 and my m1 pro updated today to 14.4 version.

Since the update, i can't access my containers (i don't think it's the os update faults, was a long time since the last reboot)

I start colima running colima start --arch aarch64 --cpu 4 --memory 4 --dns=1.1.1.1.

colima list

PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default       Running    aarch64    4           4GiB           60GiB    docker

colima version

colima version 0.6.7
git commit: ba1be00e9aec47f2c1ffdacfb7e428e465f0b58a

runtime: docker
arch: aarch64
client: v24.0.5
server: v24.0.7

docker port 09794d330269

3306/tcp -> 0.0.0.0:3306
3306/tcp -> [::]:3306

nc -zv localhost 3306

nc: connectx to localhost port 3306 (tcp) failed: Connection refused
nc: connectx to localhost port 3306 (tcp) failed: Connection refused

EDIT: downgrade to 0.5.6 made me work again :D

[alexcfpho]

Here's what I ran initially, this is for 0.6.8:

colima start --kubernetes --cpu 1 --memory 2 --disk 10 --network-address --arch aarch64 --vm-type=vz --vz-rosetta --very-verbose

The error I got was first:

> Job for k3s.service failed because the control process exited with error code.
> See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
FATA[0069] error starting kubernetes: exit status 1

I did see some issue with ssh and forwarding:

> SSH Local Port: 53847
> [hostagent] [VZ] - vm state change: running
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] The essential requirement 1 of 2 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 2 is satisfied
> [hostagent] Waiting for the guest agent to be running
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/alexpho/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/alexpho/.colima/docker.sock" (host)
> "[hostagent] guest agent events closed unexpectedly" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: unable to connect to guest agent via vsock port 2222\""
> [hostagent] Guest agent is running
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Not forwarding TCP 127.0.0.54:53
> [hostagent] Not forwarding TCP 127.0.0.53:53
> [hostagent] Not forwarding TCP [::]:22
> [hostagent] The final requirement 1 of 1 is satisfied

To resolve this, I disabled the mac os firewall and then it worked immediately. I re-enabled the firewall, and can still resolve the app in my browser and use colima ssh no problem.

Before trying this, I did downgrade to 0.5.6 as @Zordid mentioned and ran into the same issue, but the trace is slightly different; here it is for reference:

> SSH Local Port: 53684
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] [VZ] - vm state change: running
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] 2024/05/04 15:12:48 tcpproxy: for incoming conn 127.0.0.1:53688, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: connection was refused
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] 2024/05/04 15:12:58 tcpproxy: for incoming conn 127.0.0.1:53697, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: connection was refused
> [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
> [hostagent] The essential requirement 1 of 2 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 2 is satisfied
> [hostagent] Waiting for the guest agent to be running
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/alexpho/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/alexpho/.colima/docker.sock" (host)
> [hostagent] Guest agent is running
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Not forwarding TCP 0.0.0.0:22
> [hostagent] Not forwarding TCP [::]:22
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima` to open the shell.

Disabling the mac firewall resolved this issue for both versions. Hope this helps someone.

EDIT: I did have SSH as an accepted rule/app in the firewall, though I can't see how granular the ports are that are accepted there so that may be the issue—haven't looked into this further if you can specify more ports, or even need to (if mac can just review all ssh requests by protocol instead of port specified).

[jgielstra]

Reproduced the same FIX as @alexcfpho on Sonoma 14.4.1
colima start --network-address --kubernetes would fail to start

> Job for k3s.service failed because the control process exited with error code.
> See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.

Digging into journalctl, I saw IPV6 address for col0

May 06 08:37:08 colima k3s[2583]: time="2024-05-06T08:37:08-05:00" level=info msg="Found ip fdbe:ba4b:d536:7d8a:5055:55ff:fe77:b47a from iface col0"

Disabling the firewall fixed the issue; when I have time, I will need to figure out firewall settings.

After turning OFF the firewall and deleting/restarting colima, k3s starts and I have an IP assigned

 $ colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME       ADDRESS
default    Running    aarch64    6       12GiB     60GiB    docker+k3s    192.168.106.2