Export/import Docker data (migration, backup, storage location change)

[wizardcoder]

Is your feature request related to a problem? Please describe.

There is no way to export, migrate, or reliably back up Docker data in OrbStack.

I have 40+ projects running via DDEV on an external drive. I want to move to my internal drive for speed and reliability, but changing the storage location in Settings > Storage starts with a blank slate, with no way to import Obstacks docker data.

This also affects anyone moving to a new Mac (#1511), anyone whose drive fails, or anyone wanting to switch to Docker Desktop (migration only works one way).

Describe the solution you'd like

A way to export and import the full Docker state, equivalent to what already exists for Linux machines (right-click → Export). Or a migrate option when changing the storage location so data is moved rather than abandoned.

Describe alternatives you've considered

• Time Machine - restoring data.img often fails and the file can't be mounted (is there any way to open/read an data.img file from a time machine backup? #2046)
• Third-party backup tools - all struggle with the 8TB sparse file (Orbstack's data.img is shown as 8.8TB in size, throwing off Carbon Copy Cloner (and probably others) #512)
• Docker CLI - can handle individual images/volumes but not full container configs, and is very time consuming across 40+ projects
• Excluding from backups as the docs suggest - this means Docker data (databases, volumes, container state) has no backup at all. As more users experience drive failures, Mac upgrades, or simply need to change drives, they're going to discover they've permanently lost work with no way to recover it. This isn't acceptable for any professional workflow.

Additional context

Related: #249, #239, #1511, #1614, #1925, #2046, #512, [Discussion #1003](https://github.com/orgs/orbstack/discussions/1003)

[kdrag0n]

tar should have no issues with the data.img.

[tisba]

@kdrag0n is referring to the fact that the data.img (data.img.raw) files are sparse files and tar should be able to handle this properly.

Looking at tar's manpage (or GNU tar), it looks it should work. However when testing this out myself, I couldn't quite get tar to handle it properly after all:

$ du -h data.img.raw
23G    data.img.raw
$ ls -lh data.img.raw
-rw-r--r--@ 1 basti  staff   8.0T Apr 20 18:12 data.img.raw

GNU tar (1.35) on macOS does not support --sparse --hole-detection=seek which falls back to raw, meaning the entire file will be read twice. Which takes… FOREVER! For more information, see https://www.gnu.org/software/tar/manual/html_node/sparse.html.

On the other hand, tar shipped with macOS (bsdtar 3.5.3) does not seem to work properly either. It reads the file only ones, but does not account for the fact that the file is sparse.

$ tar --read-sparse -cf test.tar data.img.raw

Will start writing test.tar which quicky grows way beyond the 23G (see above).

[mpfz0r]

sparse_copy.py - Copy a sparse file using SEEK_HOLE/SEEK_DATA.

Reads only the allocated data extents of the source and writes them to
the destination at the same offsets, preserving the sparse layout.
The destination is created as a sparse file of the same apparent size
as the source.

Tested target: macOS/APFS. Should also work on Linux with sparse-aware
filesystems (ext4, xfs, btrfs).

Usage: sparse_copy.py SOURCE DEST

DEST must not exist (the script refuses to overwrite).

Notes for the OrbStack use case:

• Quit OrbStack fully before running. Verify with orb status.
• This script copies one file. The OrbStack data directory contains
  other small files alongside data.img - copy those separately with
  cp -R after.
• After the copy completes, verify integrity with:
  shasum -a 256 SOURCE DEST
  The two hashes must match exactly.
• If anything goes wrong, the original is untouched. Keep it until
  you have verified the destination works end-to-end.

sparse_copy.py

[mpfz0r]

I've had success with this script claude gave me :) the new bottleneck becomes shasum, which is single threaded perl.
You can use b3sum or the likes, but it will take a while to churn through 8TB of zeros.

I've actually skimmed the verification. YOLO :)

[nathan1658]

sparse_copy.py - Copy a sparse file using SEEK_HOLE/SEEK_DATA.

Reads only the allocated data extents of the source and writes them to the destination at the same offsets, preserving the sparse layout. The destination is created as a sparse file of the same apparent size as the source.

Tested target: macOS/APFS. Should also work on Linux with sparse-aware filesystems (ext4, xfs, btrfs).

Usage: sparse_copy.py SOURCE DEST

DEST must not exist (the script refuses to overwrite).

Notes for the OrbStack use case:

• Quit OrbStack fully before running. Verify with orb status.
• This script copies one file. The OrbStack data directory contains
  other small files alongside data.img - copy those separately with
  cp -R after.
• After the copy completes, verify integrity with:
  shasum -a 256 SOURCE DEST
  The two hashes must match exactly.
• If anything goes wrong, the original is untouched. Keep it until
  you have verified the destination works end-to-end.

sparse_copy.py

Worked for me, thanks a lot!