Problems creating x86_64 on aarch64

[dhess]

Hi, thanks for colima! I've been using it for a month or so with Nix on an M1 Mac: https://github.com/hackworthltd/hacknix/blob/main/nix/pkgs/colima/default.nix

It's been working great with 519a372, both for aarch64 and x86_64 images.

Today I bumped to 57469b9, and now I get this:

 colima start --with-kubernetes --arch x86_64 --cpu 2 --memory 8                                                                                                
INFO[0000] using docker runtime                         
INFO[0000] starting colima                              
INFO[0000] starting ...                                  context=vm
INFO[0023] provisioning ...                              context=docker
INFO[0026] starting ...                                  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> open /usr/local/colima/binfmt.tar: no such file or directory
WARN[0034] could not enable multi-arch images: exit status 1  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Unable to find image 'colima-binfmt:latest' locally
> docker: Error response from daemon: pull access denied for colima-binfmt, repository does not exist or may require 'docker login': denied: requested access to the resource
> See 'docker run --help'.
WARN[0035] could not enable multi-arch images: exit status 125  context=docker
INFO[0036] downloading and installing ...                context=kubernetes
INFO[0088] loading oci images ...                        context=kubernetes
INFO[0091] starting ...                                  context=kubernetes
INFO[0116] updating config ...                           context=kubernetes
INFO[0116] Switched to context "colima".                 context=kubernetes
INFO[0116] done                                         
(sagan) ~  colima status                                                                                                                                                  ~
INFO[0000] colima is running                            
INFO[0000] runtime: docker                              
INFO[0000] arch: aarch64                                
INFO[0000] kubernetes: enabled                          

~  colima list                                                                                                                                                    ~
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK
default    Running    aarch64    2       8GiB      60GiB

Oddly, I get similar warnings without Kubernetes support, but it does create an x86_64 image:

 colima start --arch x86_64 --cpu 2 --memory 8                                                                                                                  ~
INFO[0000] starting colima                              
INFO[0000] creating and starting ...                     context=vm
INFO[0123] provisioning ...                              context=docker
INFO[0124] restarting VM to complete setup ...           context=docker
INFO[0124] stopping ...                                  context=vm
INFO[0134] starting ...                                  context=vm
INFO[0222] starting ...                                  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0230] could not enable multi-arch images: exit status 1  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
> See 'docker run --help'.
WARN[0231] could not enable multi-arch images: exit status 125  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0233] could not clear image cache for multi-arch images: exit status 1  context=docker
INFO[0233] done                                         

 colima status                                                                                                                                                  ~
INFO[0000] colima is running                            
INFO[0000] runtime: docker                              
INFO[0000] arch: x86_64         

Edit #1

Now, after running the previous command, deleting the cluster, and then trying again with Kubernetes, I get this:

 colima start --with-kubernetes --arch x86_64 --cpu 2 --memory 8                                                                                                ~
INFO[0000] starting colima                              
INFO[0000] creating and starting ...                     context=vm
INFO[0090] provisioning ...                              context=docker
INFO[0090] restarting VM to complete setup ...           context=docker
INFO[0090] stopping ...                                  context=vm
INFO[0101] starting ...                                  context=vm
INFO[0180] starting ...                                  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0187] could not enable multi-arch images: exit status 1  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
> See 'docker run --help'.
WARN[0190] could not enable multi-arch images: exit status 125  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0193] could not clear image cache for multi-arch images: exit status 1  context=docker
INFO[0194] downloading and installing ...                context=kubernetes
INFO[0246] loading oci images ...                        context=kubernetes
INFO[0325] starting ...                                  context=kubernetes
INFO[0332] updating config ...                           context=kubernetes
FATA[0332] error starting kubernetes: error at 'updating config': error fetching kubeconfig on guest: exit status 1 

 colima status                                                                                                                                                  ~
INFO[0000] colima is running                            
INFO[0000] runtime: docker                              
INFO[0000] arch: x86_64                                 
INFO[0001] kubernetes: enabled           

I can't reproduce the original error anymore.

FYI, I'm not running Docker, and the versions that I'm using are as follows:

 docker version
Client:
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        v20.10.9
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           darwin/arm64
 Context:           colima
 Experimental:      true

Server:
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.10
  Git commit:       847da184ad5048b27f5bdf9d53d070f731b43180
  Built:            Fri Nov 19 03:41:34 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.8
  GitCommit:        1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        

 kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"darwin/arm64"}

[dhess]

FYI, with 519a372 and everything else the same as above:

 colima start --arch x86_64 --cpu 2 --memory 8                                                                                                                  ~
INFO[0000] starting colima                              
INFO[0000] creating and starting ...                     context=vm
INFO[0312] provisioning ...                              context=docker
INFO[0312] provisioning in VM ...                        context=docker
INFO[0540] restarting VM to complete setup ...           context=docker
INFO[0541] stopping ...                                  context=vm
INFO[0563] starting ...                                  context=vm
INFO[0799] starting ...                                  context=docker
INFO[0847] done                                         

[abiosoft]

@dhess there are major changes in the underlying linux distro. You would need to delete your existing instance and start afresh. i.e. colima delete

This would be part of the release notes for the next version.

Thanks.

[dhess]

FYI, all of those colima start commands were run with no existing instance running (i.e., they had been preceded by colima delete).

[abiosoft]

@dhess running cross-architecture with --with-kubernetes flag is most likely not going to work as cross-architecture is much slower.

Do you experience same behaviour with Docker and Containerd runtime (without kubernetes)?

[dhess]

Do you experience same behaviour with Docker and Containerd runtime (without kubernetes)?

519a372 works fine with plain Docker and x86_64. I can create a cluster using Kind without any problem. (It's a bit slow, but useable.)

57469b9 doesn't work at all with plain Docker and x86_64.

I haven't tested containerd.

[abiosoft]

@dhess a fix has been pushed to main branch. Can you kindly give it another try?

Thanks.

[dhess]

Thanks! I'll try this out in the next day or two.

[dhess]

On main, same problem with --with-kubernetes:

 colima start --with-kubernetes --arch x86_64 --cpu 2 --memory 8                                                  ~/git/hackworth-k8s-test
INFO[0000] starting colima                              
INFO[0000] creating and starting ...                     context=vm
INFO[0090] provisioning ...                              context=docker
INFO[0090] restarting VM to complete setup ...           context=docker
INFO[0090] stopping ...                                  context=vm
INFO[0101] starting ...                                  context=vm
INFO[0180] starting ...                                  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0187] could not enable multi-arch images: exit status 1  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
> See 'docker run --help'.
WARN[0189] could not enable multi-arch images: exit status 125  context=docker
> command-line line 0: Unsupported option "gssapiauthentication"
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0192] could not clear image cache for multi-arch images: exit status 1  context=docker
INFO[0193] downloading and installing ...                context=kubernetes
INFO[0249] loading oci images ...                        context=kubernetes
INFO[0327] starting ...                                  context=kubernetes
INFO[0334] updating config ...                           context=kubernetes
FATA[0334] error starting kubernetes: error at 'updating config': error fetching kubeconfig on guest: exit status 1 

However, the non-Kubernetes x86_64 case now works well.

[abiosoft]

Thanks, that suffices. cross-architecture is not expected to work with --with-kubernetes.