Add copy-on-write support when using MacOS.

[eth-p]

This series of commits adds support for using the clonefile syscall for copying regular files using APFS's copy-on-write mechanism.

It's implemented in a way that:

• Is opt-in, rather than opt-out.
• Has three settings:
  1. NeverCopyOnWrite (default)
  2. CopyOnWritePreferred — try to copy-on-write, fall back to copying contents if that fails
  3. CopyOnWriteRequired — require copy-on-write to succeed when the platform is supported
• Is tested.
  • I added a -tags=test_cow flag for tests so it can run all the normal tests using CopyOnWritePreferred.
  • I updated the CI matrix to run tests both with test_cow and without.
• Asks for forgiveness rather than permission.
  • The operating system is already performing validity checks.
  • It's less syscall overhead and code to try and fail copy-on-write than to check if it's possible first.

This pull request doesn't include Linux support, as I don't currently have access to a Linux machine running btrfs.
It should be fairly straightforward to use the syscalls for it, though: https://stackoverflow.com/a/52799021

Benchmarks

Environment is a M1 MacBook Pro.

Part 1: Performance Impact for Non-CoW Usage

These Benchmarks are performed by running go test -count=1 10 times.

Benchmark @ origin/main

ok      github.com/otiai10/copy 5.272s
ok      github.com/otiai10/copy 5.261s
ok      github.com/otiai10/copy 5.274s
ok      github.com/otiai10/copy 5.270s
ok      github.com/otiai10/copy 5.267s
ok      github.com/otiai10/copy 5.268s
ok      github.com/otiai10/copy 5.277s
ok      github.com/otiai10/copy 5.242s
ok      github.com/otiai10/copy 5.254s
ok      github.com/otiai10/copy 5.262s

min: 5.242s
max: 5.277s
avg: 5.2647s

Baseline.

**Benchmark @ 6bf4915 **

ok      github.com/otiai10/copy 5.256s
ok      github.com/otiai10/copy 5.276s
ok      github.com/otiai10/copy 5.256s
ok      github.com/otiai10/copy 5.269s
ok      github.com/otiai10/copy 5.252s
ok      github.com/otiai10/copy 5.273s
ok      github.com/otiai10/copy 5.280s
ok      github.com/otiai10/copy 5.276s
ok      github.com/otiai10/copy 5.266s
ok      github.com/otiai10/copy 5.259s

min: 5.252s
max: 5.280s
avg: 5.2671s

Conclusion: With the extra abstraction, the difference is negligible. If anything, it's likely noise from inconsistent IO speeds or background processes.

Part 2: Performance Impact for CoW Usage

These benchmarks are performed by running go test -run '^TestOptions_CopyOnWrite$' -count=1 10 times.
With the file size being 32 MiB, these are the results:

**Benchmark @ 6b56d09 **

ok      github.com/otiai10/copy 0.226s
ok      github.com/otiai10/copy 0.235s
ok      github.com/otiai10/copy 0.218s
ok      github.com/otiai10/copy 0.233s
ok      github.com/otiai10/copy 0.247s
ok      github.com/otiai10/copy 0.241s
ok      github.com/otiai10/copy 0.230s
ok      github.com/otiai10/copy 0.223s
ok      github.com/otiai10/copy 0.237s
ok      github.com/otiai10/copy 0.225s

min: 0.218s
max: 0.247s
avg: 0.2315s

Baseline.

**Benchmark @ c97ba43 **

ok      github.com/otiai10/copy 0.200s
ok      github.com/otiai10/copy 0.200s
ok      github.com/otiai10/copy 0.220s
ok      github.com/otiai10/copy 0.204s
ok      github.com/otiai10/copy 0.211s
ok      github.com/otiai10/copy 0.218s
ok      github.com/otiai10/copy 0.220s
ok      github.com/otiai10/copy 0.207s
ok      github.com/otiai10/copy 0.214s
ok      github.com/otiai10/copy 0.211s

min: 0.200s
max: 0.220s
avg: 0.2105s

Conclusion: A small improvement for a 32 MiB file.

I also tested a 16 GiB file, and it goes down from 8.058s to 0.381s.

[otiai10]

First of all, thank you so much for your proposal. It's very interesting.
I reviewed your commits and a few thoughts came to mind.

1. The CI is failing, which is not acceptable as in its current state.
2. We shouldn't use CopyInTest as it doesn't represents users use-cases. We should avoid any special logic in the tests as much as possible.
3. Is it the responsibility of copy's (and its option's) to decide whether or not to use copy-on-write function? If the user is aware of using copy-on-write, they should handle it with their code like unix.Clonefile.
4. What is applyPermissionControl? I don't think it is directly related to your goal. Making things too DRY here could be confusing. Let's aim to have one clear focus per pull request.

I really appreciate your contributions, and I'd like to discuss the third point of the list above in more detail.

[eth-p]

Thanks for the quick response!

To address the points you mentioned:

The CI is failing, which is not acceptable as in its current state.

Don't worry, I'm not about to introduce changes that leave the CI in a failing state :)

Looking at the failing tests, it appears that using two wrapped errors in fmt.Errorf isn't supported until Go 1.20. Depending on what you feel the better trade off is, I can think of two reasonable approaches to fixing it:

• Removing ErrNoCOW, keeping things simpler but losing the ability to check if the error was caused by copy-on-write.
• Adding a wrapper struct that implements support for errors.Is, errors.As, and errors.Unwrap

We shouldn't use CopyInTest as it doesn't represents users use-cases. We should avoid any special logic in the tests as much as possible.

Fair. I would have preferred if it were possible to pass along a defaultTestingOptions instead of wrapping the whole Copy function, but the Options structs aren't correctly merged and fixing that was out of scope for the pull request. Perhaps instead of wrapping Copy itself, I could have the function modify the Options struct instead?

func withTestDefaults(opts... Option) []Option {
    // Change the options here.
}

// And use it like:
err = Copy("src", "dest", withTestDefaults(Options { /* ... */ }))

Is it the responsibility of copy's (and its option's) to decide whether or not to use copy-on-write function?

In my opinion, copyis often used to fully copy entire directory trees (see this search), and using copy-on-write where possible is a simple yet significant optimization to that behaviour.

To explain in a bit more detail, the goal is to create a perfect copy of a file. By directly using clonefile (or the Linux equivalent ioctl), we're able to directly tell the kernel that our goal is to create a copy. The kernel has more knowledge on how the filesystem works and on how the file is stored on the physical medium, and it's able to take advantage of that knowledge to create a new file that points to the same contents as the original file. It clones/copies the file without having to spend IO time reading or writing the entire file contents. Effectively, it's an os.Link where changing the contents of one file doesn't change the other.

In contrast, the current implementation is using a dd-like approach of copying files by reading blocks of data from the source file into memory, and then writing those blocks back to a new file. It's reliable and straightforward to understand, but it's expensive and spends a lot of time waiting for I/O to complete. It's also not optimal from a memory throughput utilization or CPU standpoint¹.

If the user is aware of using copy-on-write, they should handle it with their code like unix.Clonefile.

I do think users know what "copy-on-write" is as an idea², but I don't think nearly as many of them are aware of how to actually use it in a program let alone on the command line³. Go's standard library also doesn't make it accessible as a simple function, which means that using the mechanism requires diving into OS-specific x/sys weeds.

I also feel that a lot of the utility provided by the copy package would be lost if users were told to use unix.Clonefile (or other OS equivalent) directly instead of letting copy handle it for them. This package doesn't just provide support for copying single files, but also handles the verbose and painful parts:

• Recursing directories and creating the destination directories.
• Copying file permissions.
• Copying file times.
• Filtering/skipping files.
• Handling symlinks.
• Handling other non-regular files like named pipes.

A small side note: I would actually have preferred if the copy-on-write support I added in this pull request was opt out instead of opt in, but I didn't want to make such a significant change without your feedback or a v2 package.

What is applyPermissionControl? I don't think it is directly related to your goal. Making things too DRY here could be confusing. Let's aim to have one clear focus per pull request.

Fair point. I didn't have the full history behind PermissionControl, but this was my understanding of how it was used with the pre-PR fcopy function:

1. The destination file is created.
2. PermissionControl is used to update the permissions immediately after creation.
3. The file contents are copied.
4. The file ownership and times are updated.

I refactored it into applyPermissionControl to DRY, and as an attempt to keep the above sequence no matter which implementation was used to perform the actual copying. If it's not necessary that permissions are updated before the file contents are written, I can instead remove applyPermissionControl and apply the permissions at the end of the fcopy function like with how PreserveOwner and PreserveTimes are applied.

Footnotes

1. Syscalls and context switching from user mode to the kernel is considerably more expensive than regular function calls made in userland, since they have a lot of overhead. This way of copying a file requires performing 2n/len(buffer) syscalls for reading and writing data, which is O(n) of overhead. Additionally, read and write need to copy the data from kernel memory into process memory. Linux actually has the sendfile syscall just to avoid all of that 😅 ↩

2. For example, Apple used instant file copying as a selling point push people towards accepting APFS as the new default filesystem on Macs. Or, for another example, btrfs enthusiasts. ↩

3. On Linux, you need need cp --reflink=auto. On Mac, you need cp -c. ↩

[otiai10]

Thanks.

I'll make my proposal on it and will ask you some comments on it.

I don't want you to do double work so you can wait n see, though I don't stop you ;)

[eth-p]

If you're happy with the overall structure/design I came up with here, I'm happy to take your feedback into account and update my commits. It will save you the trouble of writing an alternate implementation :)

[otiai10]

I have alternative idea for interface design.
So please stay tuned for a moment.

[eth-p]

Looking forward to seeing it!

[otiai10]

One thing to clarify. Do we really need fallback?
I don't think so.

[eth-p]

Having a fallback is needed, but it doesn't have to be an option that the user can change.

Unlike block copying, copy-on-write has a few extra requirements:

1. The src and dest must be on the same filesystem/mount point.
2. The filesystem needs to support copy-on-write.
3. The dest file can not already exist.

The fallback is needed in case of (1) and (2).

[otiai10]

Is there any case in which users CANNOT know the cases (1) and (2) beforehand?

[otiai10]

@eth-p Please check it out when you have time ;)
This is what I meant #161
Just for interface design. Let's discuss there.

[eth-p]

Is there any case in which users CANNOT know the cases (1) and (2) beforehand?

Technically, no. From a practical standpoint, I would say "yes".

It's always possible to check for individual files:

For (1), you can compare the st_dev numbers when stat()-ing files. It requires casting to a syscall.Stat_t, by the looks of it.

For (2), there are x/sys functions available to check if the filesystem supports it , but that's getting into platform-specific behavior.

It's not reasonable to expect that those always hold true for every file when it's true for the original source and destination directories, though. For example, a subdirectory under the source could be a mount point for a different filesystem, or the destination directory could already exist and have a different filesystem mounted to a subdirectory under it.

[eth-p]

After reading your version, I feel that some additional context around my implementation would be helpful:

I was going for a transparent implementation where the user could "set and forget," so to speak. They would set the option, and copy would try its best to use copy-on-write as a way to improve its speed.

Eventually, if you made a copy/v2 package, copy-on-write would become a default option or just be built-in without being an option.

[eth-p]

I'll make some changes to this after work today to simplify things. Hopefully you'll prefer that implementation over the one I made before talking with you 😄

[otiai10]

Thank you. Don't rush, take your time. Looking forward your idea. Thanks again.

---

FYI:
What I want to respect on this project are:

1. Explicitness for users
2. Controllability for users
3. Readability for us; developers

That's why most of Options are func, so that we don't have to interfere in what users want to do.
Various users will use this package in various env and context, thus let's be super careful about interface design and keep it simple and elegant.

---