Builds randomly fail on Apple silicon using bind mounts with VirtioFS enabled

[dslatkin]

I have an issue on Apple Silicon where building Rust projects will randomly fail in bind-mounted volumes when VirtioFS is enabled in Docker.

I've found it pretty easy to reproduce. First make sure VirtioFS is enabled on the Docker engine and run the container:

docker run -itv /tmp/failing-builds:/failing-builds rust

Inside the container, if you try building in the bind-mounted volume, builds will randomly fail. In this example, the first build succeeds and the second fails. I'm making sure to rm -fr target/ every time to make sure the build actually happens.

$ cd /failing-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
   Compiling failing-builds v0.1.0 (/failing-builds)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "/tmp/rustcaNiWwj/symbols.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.iyl3saptcr8eyus.rcgu.o" "-Wl,--as-needed" "-L" "/failing-builds/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-1af56df70101340f.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-2d201e9ec773e2b2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-11edaa83ea3f42ca.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-be5d62f450379333.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-48892539e9a2c4de.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-f8eb1df1397eacbf.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-2e09868052a8b2d5.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-a7e259ed513c618c.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-791329d15b754d4d.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-00676b12b0ca4248.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-77a22a58722b5fb2.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-e9ab6f5f15790e88.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-fab82cfa2048cd45.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-21f6d61e1c83d48b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-bac7f74953c2c715.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-9b674ebb751cb2fa.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-088fae249a1081ae.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-29f211c7751ad68b.rlib" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-26783adfa77949da.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.73.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.12fhivavh9xnvcwf.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.1o1irrrhnw3vyi77.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.20vbih4y81y76bbn.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.4p0dypwjfckgah0y.rcgu.o: No such file or directory
          /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.mq20epahurx3o92.rcgu.o: No such file or directory
          collect2: error: ld returned 1 exit status
          

error: could not compile `failing-builds` (bin "failing-builds") due to previous error

If you try in a normal directory, not a bind-mounted volume, building works fine.

$ mkdir ~/working-builds
$ cd ~/working-builds
$ cargo init
$ while rm -fr target/ && cargo build; do :; done
~/working-builds# cargo init
     Created binary (application) package
root@90751c34564f:~/working-builds# while rm -fr target/ && cargo build; do :; done
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Compiling working-builds v0.1.0 (/root/working-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
^C

And if you disable VirtioFS and use the old gRPC FUSE file sharing implementation in Docker, building in the bind-mounted directory then works fine as well although much slower, compare 0.94s here to 0.31s from earlier. 🙁

$ cd /failing-builds
$ while rm -fr target/ && cargo build; do :; done
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
    Finished dev [unoptimized + debuginfo] target(s) in 0.94s
   Compiling failing-builds v0.1.0 (/failing-builds)
^C

Since this only fails with VirtioFS enabled, I'm guessing it's an issue on their end. I can open an issue on their repo but I figured it might be useful to have this issue tracked here anyways until VirtioFS fixes it.

Also if there's any more Rust/Linux debugging information I could collect and provide to them, or other things I could try before opening the issue there, that would be great.

(edit) Reached out to VirtioFS people and they said it's something to do with MacOS's VIRTIO implementation. I tried submitting something via Mac's feedback assistant back in October, but it hasn't gone anywhere.