nerdctl run with -v option got stuck without any explicit errors

[toricls]

What happened

The two commands below got stuck without any explicit errors.

• lima nerdctl run -d -v $(pwd):/usr/share/nginx/html:ro -p 127.0.0.1:8080:80 nginx:alpine
• lima nerdctl run -d -v $(pwd):/usr/share/nginx/html -p 127.0.0.1:8080:80 nginx:alpine (without the ro flag)

I faced this issue while I was playing with lima-vm/lima on my M1 (Arm) Mac.

What I expected

It successfully runs an nginx container with the specified volume bind mounted. Note that I was able to run the command above without the -v option.

Steps to reproduce

• Environment: M1 (Arm) Mac mini 2020 with macOS Big Sur Version 11.5.2 (I'm using Arm Mac, but it seems it's also happening on Intel Mac according to this GitHub issue comment)
• nerdctl version (inside Lima VM): nerdctl version 0.11.1
• limactl version: limactl version 0.6.1
• Lima configuration file: Pasted in gist here. (Note that the diffs from the original default configuration file are: 1) The number of CPUs, 2) The disk size, and 3) disabling ssh. loadDotSSHPubKeys option)

Create a directory for storing config file

$ mkdir a-directory-whatever && cd $_
$ vim default.yaml

# Create plain text HTML file for nginx
$ echo "replaced nginx content is here! yay!" > index.html 

1. Create a Lima VM

$ limactl start default.yaml
? Creating an instance "default" Proceed with the default configuration
INFO[0001] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.11.1/nerdctl-full-0.11.1-linux-arm64.tar.gz" (sha256:e2c8d0417b2fb79919f22a818813c646ad7ce0e600a951b6bac98340650e4435)
INFO[0001] Using cache "/Users/xxxx/Library/Caches/lima/download/by-url-sha256/cf356d64aa826ad177396feef432a14e0df5b2bb998191eff281541d329ff12c/data"
INFO[0001] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-arm64.img"
INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img"
INFO[0002] Using cache "/Users/xxxx/Library/Caches/lima/download/by-url-sha256/c40afd2acd5f2078759e01e119168213674615fff0701d53bc2bf69e5be3bf69/data"
INFO[0002] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/xxxx/.lima/default/serial.log")
INFO[0002] SSH Local Port: 60022
INFO[0002] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0012] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0020] [hostagent] The essential requirement 1 of 4 is satisfied
INFO[0020] [hostagent] Waiting for the essential requirement 2 of 4: "sshfs binary to be installed"
INFO[0032] [hostagent] The essential requirement 2 of 4 is satisfied
INFO[0032] [hostagent] Waiting for the essential requirement 3 of 4: "/etc/fuse.conf to contain \"user_allow_other\""
INFO[0041] [hostagent] The essential requirement 3 of 4 is satisfied
INFO[0041] [hostagent] Waiting for the essential requirement 4 of 4: "the guest agent to be running"
INFO[0041] [hostagent] The essential requirement 4 of 4 is satisfied
INFO[0041] [hostagent] Mounting "/Users/xxxx"
INFO[0041] [hostagent] Mounting "/tmp/lima"
INFO[0041] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available"
INFO[0041] [hostagent] Forwarding "/run/user/505/lima-guestagent.sock" (guest) to "/Users/xxxx/.lima/default/ga.sock" (host)
INFO[0041] [hostagent] The optional requirement 1 of 2 is satisfied
INFO[0041] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed"
INFO[0041] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0041] [hostagent] Not forwarding TCP 0.0.0.0:22
INFO[0041] [hostagent] Not forwarding TCP [::]:22
INFO[0044] [hostagent] The optional requirement 2 of 2 is satisfied
INFO[0045] READY. Run `lima` to open the shell.

# Check it works
$ lima
xxxx@lima-default:/Users/xxxx/go/src/github.com/toricls/a-directory-whatever $ ls
default.yaml  index.html

2. Run nginx container

2-1. Run without the volume option (It just works!)

$ lima nerdctl run -d -p 127.0.0.1:8080:80 nginx:alpine
docker.io/library/nginx:alpine:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:859ec6f2dc548cd2e5144b7856f2b5c37b23bd061c0c93cfa41fb5fb78307ead:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:4375f141ad62ca2763d3698e81ab6c61fa2cb8dba169212bf92c845158dbafb1: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:78754800625ea92fe7b32be0754194394e84a2ec0018b037f4279822bfcf5712:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0a8cd3c047c9c865bf65fa310e119ccbd3264a4050c81d0d6b5c3fd153dcce30:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d8e1f9d1b87e92444b4e8e5dab10322aa2d66c1a226e1419048328a36fc0f716:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4d5947aec77d85cbbdc91c6bf3435095c91d516cf90af8d1394d1538d0103dcd:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:50f60b67a614a5a3025f6d389391af19154139b35d314b2aa89dcd2791c4050b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e14bcbebf464b5696ca521435ef24348933f399f49bc4c3f4bce4a6c9b6fce5b:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.6 s                                                                    total:  9.3 Mi (1.7 MiB/s)
31ae1e25373c38fcd74bddf09202fcf646136eefd45b9a6f33b6b4c185354275

$ curl -I http://localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.21.1
Date: Wed, 01 Sep 2021 08:52:02 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 06 Jul 2021 15:21:34 GMT
Connection: keep-alive
ETag: "60e474fe-264"
Accept-Ranges: bytes

2-2. Run with the volume option (It gets stuck)

$ lima nerdctl run -d -v $(pwd):/usr/share/nginx/html:ro -p 127.0.0.1:8080:80 nginx:alpine

## The above command just got stuck so I checked the running containers in another terminal window
$ lima nerdctl ps
CONTAINER ID    IMAGE    COMMAND    CREATED    STATUS    PORTS    NAMES

Without the ro flag (behaves the same)

$ lima nerdctl run -d -v $(pwd):/usr/share/nginx/html -p 127.0.0.1:8080:80 nginx:alpine

## < -- stuck here -->

2-3. Just FYI but I also ran the same command with nerdctl's --debug-full flag, and it got stuck with the message follows.

$ lima nerdctl --debug-full run -d -v $(pwd):/usr/share/nginx/html:ro -p 127.0.0.1:8080:80 nginx:alpine
DEBU[0000] rootless parent main: executing "/usr/bin/nsenter" with [-r/ -w/Users/xxxx/go/src/github.com/toricls/lima-volume-mount --preserve-credentials -m -n -U -t 746 -F nerdctl --debug-full run -d -v /Users/xxxx/go/src/github.com/toricls/lima-volume-mount:/usr/share/nginx/html:ro -p 127.0.0.1:8080:80 nginx:alpine]
## < -- stuck here -->

Without the ro flag (behaves almost same)

$ lima nerdctl run -d -v $(pwd):/usr/share/nginx/html -p 127.0.0.1:8080:80 nginx:alpine
DEBU[0000] rootless parent main: executing "/usr/bin/nsenter" with [-r/ -w/Users/xxxx/go/src/github.com/toricls/lima-volume-mount --preserve-credentials -m -n -U -t 742 -F nerdctl --debug-full run -d -v /Users/xxxx/go/src/github.com/toricls/lima-volume-mount:/usr/share/nginx/html -p 127.0.0.1:8080:80 nginx:alpine]
## < -- stuck here -->

Other note

It would be nice if anyone can point me to the right direction for debugging this issue, or let me know if we need more info for further investigation!

[aleskovets]

Having the same issue with stoplight/spectral (in case it is image-related somehow)

[AkihiroSuda]

Thanks for reporting.

Workaround

nerdctl run -v /Users:/mnt, not nerdctl run -v /Users/foo/bar:/mnt

[AkihiroSuda]

Will be fixed in PR lima-vm/lima#188 (Replace reverse SSHFS with Samba)

[AkihiroSuda]

This issue was a regression in Lima v0.6.1.
Fix will be included in Lima v0.6.3, without migrating from reverse SSHFS to Samba.
lima-vm/lima#203