cmd/go: save GOMODCACHE disk space with clonefile/reflink copies of dup files?

[bradfitz]

I noticed that I have 22,896 duplicate files in my GOMODCACHE looking only at candidates that are at the same relative path within in the same module name, but just at different versions.

In other words, many files in a module don't change between versions, say, 1.0.0 and 1.0.1.

Maybe go mod download could do clonefile (macOS, APFS, cp -c) or FICLONE (Linux, XFS/btfs/etc, cp --reflink) to save disk space when possible?

The search doesn't cost much: you don't need to sha256 all files in GOMODCACHE looking for dups. Just looking at same relative path in other versions of the same module gets almost all of the win.

And then you can detect files already sharing backing storage with F_CTNL F_LOG2PHYS_EXT on macOS or fiemap on Linux.

[gabyhelp]

Related Issues

• cmd/go: speed up 'go run' by caching binaries #33468 (closed)
• cmd/go: synchronize and remove packages duplicated in x/mod #34801 (closed)
• cmd/go: clean GOCACHE based on disk usage #29561
• cmd/go: store package metadata in build cache #31417 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

cc @golang/command-line

[prattmic]

or FIOCLONE (Linux, XFS/btfs/etc, cp --reflik)

I believe this should be FICLONE and cp --reflink.

IIUC, the module cache is read-only, so I believe hard links would also be an option? Those would support more file systems.

[bradfitz]

Whoops, braino+typo. Fixed.

IIUC, the module cache is read-only, so I believe hard links would also be an option? Those would support more file systems.

Yes, but that scares me a bit, because then if somebody modifies a file in gomodcache, it does even more damage.

I've accidentally modified GOMODCACHE files plenty of times just clicking around in my editor (jumping to definition) which seems to have no problem changing permissions to in order to save files in the GOMODCACHE directory.

[Jorropo]

I have a script which does this after the fact: https://github.com/Jorropo/thunderdup/

The completely fool proof way to do so is to write the destination file then call FIDEDUPERANGE with the previous copy.
You just wrote the destination so it can't be dirty by someone doing something they should not in the GOMODCACHE or a previous bug leaving it in a bad state.
Then FIDEDUPERANGE does an in-kernel atomic block compare and only dedup ranges if the content is identical.

This only save on disk space, isn't faster since you still write once first and not bandwidth since you don't trust the existing content but it will never go wrong*.
*assuming the kernel is behaving correctly which it has been doing for years.

[mateusz834]

or FICLONE (Linux, XFS/btfs/etc, cp --reflink)

Maybe hard links instead? So that it works on ext4 (i guess FICLONE does not work there, since it is not a COW fs)

EDIT: already mentioned here #75851 (comment)

[bysiber]

Great idea using macOS clonefile for GOMODCACHE disk savings. Copy-on-write semantics would be perfect for the read-only nature of module cache files.

This is exactly the kind of OS-specific optimization that can save significant disk space. For macOS developers who also want to monitor their Go cache alongside other dev tools (Docker, Xcode DerivedData, node_modules, Homebrew, etc.), ClearDisk is a free, open-source menu bar app that monitors 44+ developer cache paths in real-time with color-coded alerts.