v1.0 roadmap: change the default mount driver from reverse-sshfs to 9p (for QEMU) and virtiofs (for vz)

[AkihiroSuda]

https://github.com/lima-vm/lima/blob/master/docs/mount.md

Lima v1.0 will change the default mount driver from reverse-sshfs to 9p for QEMU, virtiofs for vz. (The default vm driver will be vz)
RHEL-like templates will continue to use reverse-sshfs as their kernel lacks support for virtio-9p-pci.

[afbjorklund]

Did changing the sftp server fix the odd bugs, like when you can't compile any code (due to umask) ?

• Support using OpenSSH sftp server instead of using github.com/pkg/sftp sshocker#17

• Compiling go code, inside host mounted folder prevent the binary file being executable #602

Support for virtfs is also lacking on Windows and other host OS (like BSD), requires Linux or macOS

[AkihiroSuda]

QEMU 7.1 is released and seems to have resolved issues like:

• "Network dropped connection on reset" errors on 9p mounts #831

@jandubois @afbjorklund Are we ready to change the driver (on Linux and macOS) and release v1.0 soon?

[afbjorklund]

According to one user*, there might be 9p support also on Windows hosts soon: (haven't tried it myself, not included in 7.1.0)

* #909 (comment)

As far as I know, there should be no major issues with virtfs on Linux (if reverse-sshfs is still available as a fallback, that is) ?

[AkihiroSuda]

Before changing the default driver probably we should add a simple CLI flag like limactl start --mount-type=reverse-sshfs to allow switching back to the reverse-sshfs driver. (Colima already has a similar flag)

The --mount-type=... flag value will be stored in the "local" override.yaml stored in the instance dir so as not to break comment lines and white spaces in the main lima.yaml.
The "global" override.yaml in the _config dir will still have higher precedence.
I'll work on this after getting #1069 merged.

(Instead of messing up the overriding system we could just modify lima.yaml, but modifying YAMLs without breaking comments lines and white spaces turned out to be hard: https://github.com/AkihiroSuda/yamlctl)

cc @jandubois

[dnrce]

Per #454 (reply in thread), will a writable ~ mount also become the default as part of this change?

[XVilka]

Recently merged (in maintainer tree) 9pfs patch could improve the speed significantly, which will obviously help lima as well:

• cschoenebeck/qemu@8ab70b8
• https://linus.schreibt.jetzt/posts/qemu-9p-performance.html

[AkihiroSuda]

Recently merged (in maintainer tree) 9pfs patch could improve the speed significantly, which will obviously help lima as well:

• cschoenebeck/qemu@8ab70b8
• https://linus.schreibt.jetzt/posts/qemu-9p-performance.html

👍

I guess we should change the default to 9p after QEMU 7.2 (?) gets released with this commit

[abiosoft]

I guess we should change the default to 9p after QEMU 7.2 (?) gets released with this commit

There are still two issues to take note of, symlinks and file permissions for rootless containers.

1. Symlinks

AFAIK they are still not supported.

It used to be a non-issue for me until I switched from Homebrew to Nix. Then I quickly realised this may be a deal-breaker for some, as their environments may rely on symlinks.

• Symlinks created on the host are not accessible on the VM
• Symlinks created on the VM are not accessible on the host
• Symlinks created on the VM is accessible to the VM

2. File permissions for containers

This is not an issue with Lima but with containers running on a Lima VM.

• Containers running as non-root must use a user with same uid as the host, otherwise files are not readable.
• Some docker images assume uid 100 (e.g. official nodejs docker image), mounted files are thereby not readable.
• The behaviour is actually accurate with 9p volumes IMO. However, it appears broken due to users being used to the behaviour of Docker Desktop and are usually migrating from Docker Desktop.
• Docker Desktop (and sshfs volumes) permits read access to all mounted files regardless of container user's uid.
• An example scenario is here with vscode devcontainers pre 0.4.0 abiosoft/colima#251 (comment)

[AkihiroSuda]

Some docker images assume uid 100 (e.g. official nodejs docker image), mounted files are thereby not readable.

Seems readable (although not writable) for me:

nerdctl run --rm -v /Users/$(whoami):/mnt --user 100 alpine cat /mnt/.bashrc

[AkihiroSuda]

Lack of support for symlinks (with mapped mode) seems by design 😞

https://gitlab.com/qemu-project/qemu/-/issues/173

[AkihiroSuda]

For supporting symlinks, shall we change the default securityModel from mapped-xattr to none?
https://wiki.qemu.org/Documentation/9psetup

This is obviously incompatible with chmod, but it is not supported by the reverse-sshfs driver either, so this is probably fine.
Supporting symlinks seems more important.

security_model=mapped-xattr|mapped-file|passthrough|none: Specifies the security model to be used for this export path. Security model is mandatory only for "local" fsdriver. Other fsdrivers (like "proxy") don't take security model as a parameter. Recommended option is "mapped-xattr".

1. passthrough: Files are stored using the same credentials as they are created on the guest. This requires QEMU to run as root.
2. mapped: Equivalent to "mapped-xattr".
3. mapped-xattr: Some of the file attributes like uid, gid, mode bits and link target are stored as file attributes. This is probably the most reliable and secure option.
4. mapped-file: The attributes are stored in the hidden .virtfs_metadata directory. Directories exported by this security model cannot interact with other unix tools.
5. none: Same as "passthrough" except the sever won't report failures if it fails to set file attributes like ownership (chown). This makes a passthrough like security model usable for people who run kvm as non root.

[XVilka]

Lack of support for symlinks (with mapped mode) seems by design 😞

https://gitlab.com/qemu-project/qemu/-/issues/173

I recommend asking this again in qemu-devel mailing list since it has much more attention from developers.

[abiosoft]

Seems readable (although not writable) for me

Really? what's the permission you've got on the file on your host filesystem?

It doesn't work for me, only works as root or 501 (my macOS host uid).

# attempt uid 100, failed
$ nerdctl -- run --rm -v /Users/$(whoami):/mnt --user 100 alpine cat /mnt/.bashrc
cat: can't open '/mnt/.bashrc': Permission denied
FATA[0000] exit status 1

# attempt uid 0, success
$ nerdctl -- run --rm -v /Users/$(whoami):/mnt alpine cat /mnt/.bashrc
[ -f ~/.fzf.bash ] && source ~/.fzf.bash

# attempt uid 501, success
$ nerdctl -- run --rm -v /Users/$(whoami):/mnt --user 501 alpine cat /mnt/.bashrc
[ -f ~/.fzf.bash ] && source ~/.fzf.bash

For supporting symlinks, shall we change the default securityModel from mapped-xattr to none?
https://wiki.qemu.org/Documentation/9psetup

I can verify that symlink works with none attribute, however it breaks chmod like you said 😞

[AkihiroSuda]

Really? what's the permission you've got on the file on your host filesystem?

Yes.

$ sw_vers -productVersion
12.6

$ uname -srm
Darwin 21.6.0 x86_64

$ qemu-system-x86_64 -version
QEMU emulator version 7.1.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

$ limactl -v
limactl version 0.12.0

$ limactl start template://experimental/9p 

$ id
uid=501(suda) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1)

$ ls -ln .bashrc 
-rw-r--r--  1 501  20  139 10  6 10:24 .bashrc

$ limactl shell 9p ls -ln /Users/$(whoami)/.bashrc
-rw-r--r-- 1 501 20 139 Oct  6 01:24 /Users/suda/.bashrc

$ limactl shell 9p -- nerdctl run --rm -v /Users/$(whoami):/mnt --user 100 alpine cat /mnt/.bashrc
# (My bashrc shows up)

[abiosoft]

Maybe someone else can share their experience (@jandubois maybe?). The only difference between our environments is I am testing on an m1 device and you're using an intel device.

For me, no luck still. No read access for users other than root and host uid.

[arixmkii]

According to one user*,

Seems like a good time to join this party.

This patch https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg04073.html was published recently. I added it to HEAD and built from sources and managed to use 9pfs mount with QEMU Podman machine on Windows (R/O only, but I had time only for one try). Still, not clear if this will make its way before 7.2.0 is cut.

[XVilka]

7.2-rc0 is out, it makes sense to start testing, while there is a possibility for a quick bug fix if anything is found.

[AkihiroSuda]

In abiosoft/colima#544 (comment) it was reported that 9p is still slow with QEMU 7.2 😞

[AkihiroSuda]

For v1.0, probably we will make vz and virtiofs default

• Roadmap for moving vz out of experimental #1610

[ryancurrah]

We've been using VZ and virtiofs and no issues so far.

[AkihiroSuda]

I want to plan this before KubeCon North America (Nov 6-9)

cc @lima-vm/maintainers

[afbjorklund]

As long as sshfs is still available, I don't see any issues with changing the default