Virtualised gix fails with No such device because of MAP_SHARED mmap.

[gl-yziquel]

Current behavior 😯

When using gix in a virtualised environment, on a virtiofs filesystem backed by ZFS on the host, with cache disabled (--cache=never, which matters as otherwise the situation is not reproducible), it becomes impossible to run "gix clone", whereas "git clone" works fine. The (likely) reason is the use of MAP_SHARED in an mmap system call, which is a MAP_PRIVATE in the corresponding system call for git. MAP_SHARED requires more stringent coherency requirement, which, coupled with --cache=never for virtiofs, makes the "gix clone" command fail with a "No such device" OS level error in the guest OS. Details of the strace analysis may be found here:

cross-rs/cross#1313 (comment)

Expected behavior 🤔

"gix clone" should work. The current implementation deviates from that of "git" in its use of MAP_SHARED instead of MAP_PRIVATE. On the other hand, MAP_SHARED seems to be prefered in the rust world as MAP_PRIVATE has some ugly corner cases with "unspecified behaviour".

Either gix want to be a bug-for-bug replacement for git, in case use of MAP_PRIVATE is meaningful, and would make it work in a virtualised setting.

Either gix want to stick with a safe rust focused ideology, in which case MAP_PRIVATE is to be frowned upon, but that would kick the can down the road to the question of memory mapping of files in QEMU/VirtioFS, and involves likely uncanny DAX window behaviour, that seems not to be evenly supported by virtualisation frameworks.

This is, in the end, the choice of the author of gix.

Git behavior

It works.

I personally, would like to know what line of code in the gix source code makes the call to a MAP_SHARED mmap. As that would allow to debug the situation more closely w/r to virtualisation and virtiofs (which is my real concern).

Steps to reproduce 🕹

1. Set up an incus/lxd virtual machine (not container) running Ubuntu Mantic, with the latest (daily zabbly build of today, which uses the --cache=never option when calling virtiofsd) incus from zabbly.
2. Mount a ZFS filesystem on the host
3. Export the ZFS filesystem into the guest via "incus device add" commands. (Uses virtiofs v1.10.1 under the hood)
4. In the guest, gix clone a git repository.
5. Observe the "No such device" os error 19 pop up.

[Byron]

Thanks a lot for letting me know!

gitoxide crates use the mmap2 crate which hides these details entirely. That's great as it means there is no deeper reason behind the MAP_SHARED mode, and it shouldn't be a problem to switch to MAP_PRIVATE.

Since I probably won't be able to reproduce the issue I wonder if you'd be willing to validate a version from a PR that I could create rather swiftly?

Thanks for your help.

[gl-yziquel]

Thanks a lot for letting me know!

gitoxide crates use the mmap2 crate which hides these details entirely. That's great as it means there is no deeper reason behind the MAP_SHARED mode, and it shouldn't be a problem to switch to MAP_PRIVATE.

Since I probably won't be able to reproduce the issue I wonder if you'd be willing to validate a version from a PR that I could create rather swiftly?

Thanks for your help.

I'll have to build the PR myself on that guest. Which will be a problem since I cannot run "cargo build" (for likely the same reason, which indeed led me down here first with gix). So it will take some amount of tweaking to get it right. Moreover, I have to go get some food. So I won't reply in a quick feedback loop. Do not expect a reply before sunday or monday...

[Byron]

Thanks for the quick response and for your help.

I am sorry to hear cargo also bails out in that setup. In case, for some reason, it's not due to memmap but to C toolchains or such, you should be able to build gix with cargo build --bin gix --no-default-features --features max-pure, which requires only a Rust compiler and a linker and Cargo.

[gl-yziquel]

Just looked at the proposed fix. Yes, the memmap2 crate has a way to enforce MAP_PRIVATE. I hope this will fix it (as it will allow me to narrow down the behaviour of virtiofs). Going to get some food. Then I'll try it out.

Thank you for being so unbelievably responsive.

[Byron]

Bugs get the priority lane :).

[gl-yziquel]

Just wait for the test for some 24 to 48 hours.

[gl-yziquel]

The test is conclusive: the fix you proposed works on such a virtualised setting. You should feel free to commit it to the main branch.

All clear.