Builds randomly fail on Apple silicon using bind mounts with VirtioFS enabled

[dslatkin]

I have an issue on Apple Silicon where building Rust projects will randomly fail in bind-mounted volumes when VirtioFS is enabled in Docker.

I've found it pretty easy to reproduce. First make sure VirtioFS is enabled on the Docker engine and run the container:

docker run -itv /tmp/failing-builds:/failing-builds rust

Inside the container, if you try building in the bind-mounted volume, builds will randomly fail. In this example, the first build succeeds and the second fails. I'm making sure to rm -fr target/ every time to make sure the build actually happens.

$ cd /failing-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
   Compiling failing-builds v0.1.0 (/failing-builds)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "/tmp/rustcaNiWwj/symbols.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.iyl3saptcr8eyus.rcgu.o" "-Wl,--as-needed" "-L" "/failing-builds/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-1af56df70101340f.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-2d201e9ec773e2b2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-11edaa83ea3f42ca.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-be5d62f450379333.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-48892539e9a2c4de.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-f8eb1df1397eacbf.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-2e09868052a8b2d5.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-a7e259ed513c618c.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-791329d15b754d4d.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-00676b12b0ca4248.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-77a22a58722b5fb2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-e9ab6f5f15790e88.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-fab82cfa2048cd45.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-21f6d61e1c83d48b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-bac7f74953c2c715.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-9b674ebb751cb2fa.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-088fae249a1081ae.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-29f211c7751ad68b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-26783adfa77949da.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o: No such file or directory
          collect2: error: ld returned 1 exit status
          

error: could not compile `failing-builds` (bin "failing-builds") due to previous error

If you try in a normal directory, not a bind-mounted volume, building works fine.

$ mkdir ~/working-builds
$ cd ~/working-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
~/working-builds# cargo init
     Created binary (application) package
root@90751c34564f:~/working-builds# while rm -fr target/ && cargo build; do :; done
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
^C

And if you disable VirtioFS and use the old gRPC FUSE file sharing implementation in Docker, building in the bind-mounted directory then works fine as well although much slower, compare 0.94s here to 0.31s from earlier. 🙁

$ cd /failing-builds
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
^C

Since this only fails with VirtioFS enabled, I'm guessing it's an issue on their end. I can open an issue on their repo but I figured it might be useful to have this issue tracked here anyways until VirtioFS fixes it.

Also if there's any more Rust/Linux debugging information I could collect and provide to them, or other things I could try before opening the issue there, that would be great.

(edit) Reached out to VirtioFS people and they said it's something to do with MacOS's VIRTIO implementation. I tried submitting something via Mac's feedback assistant back in October, but it hasn't gone anywhere.

[dslatkin]

I got an update from Apple on the issue I submitted with Feedback Assistant. They marked it as "Potential fix identified - For a future OS update" and asked that I test with iOS 18 beta available at https://developer.apple.com/download/.

I think this was a copy pasta mistake and they actually wanted me to test with the MacOS 15 beta. So I installed the beta, made sure Docker was up-to-date, enabled VirtioFS, and the issue no longer presents itself.

So I think once MacOS 15 is officially released, this issue can be closed.

[cynecx]

@dslatkin Have you actually tested this on a recent docker-for-mac release before you did your tests on macOS 15? I think back in February there was a docker-for-mac release that included an in-kernel (vm) fix that workarounds the bug in virtualization.framework's virtiofs implementation. I am just wondering because there might be a possibility that this bug hasn't actually been fixed yet but you're just seeing the effects of the workaround the docker devs did some months ago.

[dslatkin]

I thought I had tested somewhat recently (probably April or so). If this synchronized file store released in February is what you're thinking of, I did test a couple days later on the free version for the heck of it, and it didn't work then either.

If you're on macOS 14 with Apple silicon, you could check yourself:

docker run -itv /tmp/failing-builds:/failing-builds rust
cd /failing-builds
while rm -fr target/ && cargo build; do :; done

[cynecx]

@dslatkin Almost, you are one release off 😅 I just looked it up again and it turns out that they shipped this with 4.28 (docker/for-mac#7059 (comment)). I also have that one installed locally on my M1 and tried to do cargo builds in a loop as you said. It’s been running for one minute or so without showing any issues 🤷‍♂️.

[dslatkin]

@cynecx Ah that's good to know, thanks for sharing. Funny that the release notes for 4.27 explicitly mentioned this issue as fixed but nobody ended up pinging this thread.

I suppose the way to properly test this then would be to downgrade to 4.27, confirm if macOS 15 fixes the issue on that old version, and if fixed probably compare performance/build times and open an issue to remove the hotfix in 4.28. Right now I'm in the middle of a crunch and can't risk disrupting my working setup, but I am hopeful I can get to this in the next few weeks to test if nobody else does first.