Xcode Compilation Cache not reused across CI runs

[DominikMoreira]

Hi everyone 👋

I’m seeing unexpected behavior with Xcode’s Compilation Cache when building in a CI setup using tart and was wondering if anyone else has experienced something similar. I'm trying to understand whether this might be related to virtualization or something specific to how tart manages VM state.

Setup

• Jenkins runs on a macOS build node
• Jenkins starts a tart VM
• The source code is mounted into the VM

  tart run --root-disk-opts=sync=none,caching=cached --dir=/Users/Shared/Developer/Jenkins/workspace/:tag=build VM-Xcode264

• Build commands are executed via:

  tart exec

---

Behavior

Within a single Jenkins run:

• First build → cache miss
• Second build → cache hit ✅

Across Jenkins runs (same VM image, same workspace, same DerivedData path):

• First build → cache miss ❌
• Second build → cache hit ✅

---

Additional details

• DerivedData path is stable and reused (inside the VM)
• No changes to source code, headers, or build configuration
• CAS key is identical across runs

---

Question

Has anyone seen similar behavior when using tart VMs, where build caches (like Xcode’s compilation cache / LLVM CAS) are not reused across runs?

Could this be related to:

• how tart handles VM snapshots or disk state
• or something around filesystem/mount behavior

---

Any insights or similar experiences would be greatly appreciated!

[DominikMoreira]

After some debugging, I found the most plausible cause of the cache invalidation across VM restarts, but all common workarounds have failed so far.

The cause: Changing virtiofs Device IDs

Xcode's build system relies on file stat data, specifically the Device ID and Inode. I ran stat -x on a source file before and after a VM restart, and it turns out the virtiofs mount gets a new Device ID on every tart run.

Here is my stat -x tracking:
Build A (Initial run): Device: 53,2 -> Cache Miss
Build A (Re-run, VM kept running): Device: 53,3 -> Cache Hit ✅
Build B (VM restarted): Device: 48,2 -> Cache Miss ❌

Because the Device ID changes (e.g., from 53,x to 48,x), I guess Xcode assumes the filesystem or drive has changed and completely invalidates the cached object files.

Attempted Workarounds (that failed)

• Xcode Flag: I tried injecting defaults write com.apple.dt.XCBuild IgnoreFileSystemDeviceInodeChanges -bool YES before the build. Xcode seems to ignore this flag when dealing with a virtiofs mount.

• Local Workspace Sync: I tried copying the source code away from the virtiofs mount to a local APFS directory inside the VM before building (rsync -a --delete --modify-window=1 /virtiofs/path /local/path). The cache still misses across VM restarts.