Tart images getting deleted

[ruimarinho]

Hi,

I recently updated from 0.19.1 to 0.26.1 and I'm experiencing an issue where launching new VMs automatically deletes all tart images that have been downloaded previously.

I am not sure if this is related, but I found this on the orchard logs:

{"level":"warn","ts":1733854540.791293,"msg":"'tart fqn ghcr.io/cirruslabs/macos-runner:sonoma' failed with exit code 1: OCI storage error: ghcr.io/cirruslabs/macos-runner:sonoma is not a digest and doesn't point to a digest","vm_uid":"ea98168a-1f73-4647-9a38-f59a2043c97e","vm_name":"01JERW6WQXHPWC9DCKS3EX91FN","vm_restart_count":0}

There has been no change in terms of how VMs are launched or how the image is configured. Is this a known issue?

Thank you.

[edigaryev]

Hi Rui 👋

Orchard shouldn't normally delete VM images in the OCI cache (such as ghcr.io/cirruslabs/macos-runner:sonoma).

How much free space do you have available on your host machines, and how big are the VM images?

This behavior could be due to the automatic pruning in Tart.

As for the tart fqn error, it looks like the VM image in the OCI cache had been removed partially, i.e. only the sha256: directory was removed for some reason, leaving sonoma tag directory hanging and pointing to sha256: 🤔

[ruimarinho]

I would say around 100GB free, maybe a little bit less. It’s definitely due to the size of the instance, as the ones with 1TB do not exhibit this behavior.

I believe this is due to the runner image growing in size over time, but I think auto pruning is a bit too aggressive. Is there any way of having orchard respect that environment variable to disable auto pruning?

[edigaryev]

Is there any way of having orchard respect that environment variable to disable auto pruning?

Are you running Orchard Workers using launchd by a chance?

If so, you can just define a EnvironmentVariables block with TART_NO_AUTO_PRUNE environment variable.

This should work since Orchard passes its own environment variables to the Tart processes it spawns.

[ruimarinho]

Yep, I am using launchd. I used the suggested plist block with an empty string value and it worked as expected — thank you!

I've noticed pruning is called on every clone, which matches the behavior I am observing: once the VM is launched by tart, its image gets deleted if the available storage space is not within the expected range.

However, looking at the amount of free storage available on this machine - around 180GB - it seems like APFS is playing some tricks on https://github.com/cirruslabs/tart/blob/31ab4218f7ce73a40c3304db3137cb430cd53823/Sources/tart/Commands/Clone.swift#L77..

[edigaryev]

it seems like APFS is playing some tricks on https://github.com/cirruslabs/tart/blob/31ab4218f7ce73a40c3304db3137cb430cd53823/Sources/tart/Commands/Clone.swift#L77

Interesting observation! We probably want to calculate this a bit differently.

I've created cirruslabs/tart#973 to address this.