Add copy-on-write support for Linux BTRFS filesystem

[bozaro]

Use btrfs copy-on-write feature for copy lfs files from .git/lfs/objects to working copy (git lfs checkout).
Copy-on-write operation is look like hardlink, but change in one file is not visible in another file.

This command works like: cp --reflink=auto src.txt dst.txt
For this change I use https://github.com/wertarbyte/coreutils/blob/master/src/copy.c as reference.

[technoweenie]

This looks good. However, as someone that doesn't use linux much and has 0 experience with the BTRFS, I'll have to rely on you to support this. Based on our previous interactions, I don't think that'll be a problem, but I wanted to make that clear :)

This doesn't work when I build the linux binaries:

Building for linux/amd64
# github.com/github/git-lfs/lfs
lfs/util.go:48: undefined: CloneFile

Probably because I'm building cross platform binaries without cgo. Should be easy to fix with another file with a build flag like +build linux,!cgo or something. Though, I imagine most linux devs install from the packages, which build git-lfs from source. Part of me wonders if I should even bother with the cross compiled tarballs then?

What will this do on a linux system without btrfs? Will that syscall just return an error, causing CopyWithCallback() to copy the file manually?

[bozaro]

I write wrong build condition:
//+build !linux,cgo ((not linux) and cgo)

It's also means: you build without cgo

What will this do on a linux system without btrfs? Will that syscall just return an error, causing CopyWithCallback() to copy the file manually?

Btrfs is supported on linux a long time (for example Ubuntu 11.04 supports btrfs filesystem). In situation with unknown syscall it simply return error and failback to block-by-block copy. Failback to block-by-block copy also will be in any situations when copy-on-write is inpossible (files on different filesystems, files on ext4, too old btrfs implementation without copy-on-write and etc).

In benefits btrfs users have:

• Cloning files from .git/lfs/objects to working directory is extremely fast;
• Unmodified LFS files in working directory don't use aditional disk space.

All other user have penalty by one redundant syscall per file.

I checked this code locally on Ubuntu 14.04 ext4 and btrfs.

Personally, I'm most upset me, I do not know how to implement a similar function under Windows :(

[technoweenie]

Cool, thanks for fixing the build flag. This should make it into a new release next week. I want to make sure my linux and windows vm tests all pass first.

Thanks!

[technoweenie]

This fails in the centos 5 docker build.

$ ./docker/run_dockers.bsh centos_5
centos_5: Pulling from andyneff/git-lfs_dockers
Digest: sha256:e75cba3fb67d5b2a3490ea800bd50d15543e7529358367f1def9bf2e68cc18d3
Status: Image is up to date for andyneff/git-lfs_dockers:centos_5
Compiling LFS in docker image centos_5
...
+ ./script/bootstrap
lfs/util_linux.go
# github.com/github/git-lfs/lfs
/usr/bin/ld: unrecognized option '--build-id=none'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status

It builds successfully in all of the other docker files:

$ ./docker/run_dockers.bsh centos_{6,7} debian_{7,8}

It looks like we can pass custom build flags. Would it be worth trying to pass flags like lfs_centos_5? Then we can set the build flags for the following files:

# this contains the btrfs code
# lfs/util_linux.go
// +build linux,cgo
// +build !lfs_centos_5

# lfs/util_generic.go
// +build !linux !cgo lfs_centos_5

Though, I'm not even sure that would fix that ld error above.

[andyneff]

If I remember correctly, golang doesn't like having a build ids in the debug symbols. This is basically a checksum of a portion of the binary data (Not all of it so that if you compile the same code twice you get the same checksum). This is important in debugging in Fedora operating systems because this is how debugging symbols are matched with binary files after being turned into RPMs, where the debug symbols and binary files are split into two. In the end, I gave up and just disabled the debug symbol handling in the RPMs.

I'm guessing something in the // flags triggers go to add the --build-id flag, which Centos 5 doesn't support.

Since go doesn't really support Centos 5... This might be a good excuse to drop the Centos 5 building... ;)

[bozaro]

About this build issue
I think, this error will occurs on any cgo usage. So, simplest way is disable cgo on CentOS 5 by environment variable:

export CGO_ENABLED=0

About CentOS 5
https://golang.org/doc/install#requirements

CentOS/RHEL 5.x not supported

https://wiki.centos.org/FAQ/CentOS5#head-a2fd35a09ebf776661ab0f1288762329bb64209c

We intend to support CentOS 5 until Mar 31st, 2017 The current plan is this:

• Full Updates (including hardware updates): Currently to Q4, 2012
• Updates ( including minor hardware updates): Up to Q1 of 2014
• Maintenance Updates Q1, 2011 - Mar 31st, 2017

[andyneff]

Is cgo not needed for this BTRFS feature?

[bozaro]

I use cgo for calculate correct BTRFS_IOC_CLONE value (I'm not sure that _IOW macro is platform-independent).
Disabling cgo also turned off the change of this PR.

[technoweenie]

I'm in favor of disabling CGO on centos 5. If it's worth just dropping all support, that works too :) Once that happens, we can merge this PR.

[technoweenie]

We just ended up disabling CGO in centos 5:

• Disable CGO for Centos 5 to assist #952 #976
• disable cgo on centos 5 build-dockers#1

We'll drop centos 5 support once it's no longer under "maintenance support".

Thanks again for the patch!