os: if rename system call fails, try ioctl(FICLONE)

[codeaholics]

What version of Go are you using (go version)?

$ go version
go version go1.15.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/danny/Library/Caches/go-build"
GOENV="/Users/danny/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/danny/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/danny/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/danny/workspace/media-sort/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/36/kj75t85j1m57cg047cr5dh4m0000gn/T/go-build628346493=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I am cross-compiling on Mac for Linux

Linux execution env: Linux nas 4.4.190.x86_64.1 #1 SMP Mon Oct 28 01:55:46 UTC 2019 x86_64 GNU/Linux

Build command: GOOS=linux GOARCH=amd64 go build -v

func move(src, dst string) error {
	err := os.Rename(src, dst)
	// cross device move
	if err != nil && strings.Contains(err.Error(), "cross-device") {
		if err := copy(src, dst); err != nil {
			return err
		}
		if err := os.Remove(src); err != nil {
			return err
		}
	}
	return nil
}

What did you expect to see?

os.Rename() to succeed because the source and target are on the same filesystem.

What did you see instead?

os.Rename() returns "invalid cross-device link" and the above code falls back to copy-and-delete. If I do an mv on the command line, it completes immediately - even for multi-gigabyte files.

[ianlancetaylor]

os.Rename is a fairly thin wrapper around syscall.Rename, which just calls the rename system call. If possible, please run the final GNU/Linux executable under strace -f and see what the rename system call returns.

[codeaholics]

Thanks for your quick response @ianlancetaylor. To test, I made an empty file, f, in /data/Videos on my NAS, and mv-ed it to /data/Music. Here's the strace:

execve("/bin/mv", ["mv", "f", "/data/Music"], [/* 17 vars */]) = 0
brk(0)                                  = 0x16eb000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d8000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=28370, ...}) = 0
mmap(NULL, 28370, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe1406d1000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20c\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=142728, ...}) = 0
mmap(NULL, 2246896, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe140295000
mprotect(0x7fe1402b6000, 2097152, PROT_NONE) = 0
mmap(0x7fe1404b6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7fe1404b6000
mmap(0x7fe1404b8000, 6384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe1404b8000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\37\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=35288, ...}) = 0
mmap(NULL, 2130592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe14008c000
mprotect(0x7fe140094000, 2093056, PROT_NONE) = 0
mmap(0x7fe140293000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fe140293000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18640, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d0000
mmap(NULL, 2113912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13fe87000
mprotect(0x7fe13fe8b000, 2093056, PROT_NONE) = 0
mmap(0x7fe14008a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fe14008a000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1738176, ...}) = 0
mmap(NULL, 3844640, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13fadc000
mprotect(0x7fe13fc7d000, 2097152, PROT_NONE) = 0
mmap(0x7fe13fe7d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a1000) = 0x7fe13fe7d000
mmap(0x7fe13fe83000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe13fe83000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=448440, ...}) = 0
mmap(NULL, 2543976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f86e000
mprotect(0x7fe13f8da000, 2097152, PROT_NONE) = 0
mmap(0x7fe13fada000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c000) = 0x7fe13fada000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406cf000
mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f66a000
mprotect(0x7fe13f66d000, 2093056, PROT_NONE) = 0
mmap(0x7fe13f86c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fe13f86c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320n\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137384, ...}) = 0
mmap(NULL, 2213008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f44d000
mprotect(0x7fe13f465000, 2093056, PROT_NONE) = 0
mmap(0x7fe13f664000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fe13f664000
mmap(0x7fe13f666000, 13456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe13f666000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406ce000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406cc000
arch_prctl(ARCH_SET_FS, 0x7fe1406cc800) = 0
mprotect(0x7fe13fe7d000, 16384, PROT_READ) = 0
mprotect(0x7fe13f664000, 4096, PROT_READ) = 0
mprotect(0x7fe13f86c000, 4096, PROT_READ) = 0
mprotect(0x7fe13fada000, 4096, PROT_READ) = 0
mprotect(0x7fe14008a000, 4096, PROT_READ) = 0
mprotect(0x7fe140293000, 4096, PROT_READ) = 0
mprotect(0x7fe1404b6000, 4096, PROT_READ) = 0
mprotect(0x61e000, 4096, PROT_READ)     = 0
mprotect(0x7fe1406da000, 4096, PROT_READ) = 0
munmap(0x7fe1406d1000, 28370)           = 0
set_tid_address(0x7fe1406ccad0)         = 23496
set_robust_list(0x7fe1406ccae0, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7fe13f4539b0, [], SA_RESTORER|SA_SIGINFO, 0x7fe13f45c890}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fe13f453a40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fe13f45c890}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffc57d353f0) = -1 ENOENT (No such file or directory)
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=965552, f_bfree=722015, f_bavail=665542, f_files=1048576, f_ffree=1032627, f_fsid={-1827525020, -625743731}, f_namelen=255, f_frsize=4096}) = 0
brk(0)                                  = 0x16eb000
brk(0x170c000)                          = 0x170c000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d7000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tt"..., 1024) = 370
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fe1406d7000, 4096)            = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=6839968, ...}) = 0
mmap(NULL, 6839968, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe13edc7000
close(3)                                = 0
geteuid()                               = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("/data/Music", {st_mode=S_IFDIR|0777, st_size=38, ...}) = 0
lstat("f", {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
lstat("/data/Music/f", 0x7ffc57d35060)  = -1 ENOENT (No such file or directory)
rename("f", "/data/Music/f")            = -1 EXDEV (Invalid cross-device link)
unlink("/data/Music/f")                 = -1 ENOENT (No such file or directory)
open("f", O_RDONLY|O_NOFOLLOW)          = 3
fstat(3, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
open("/data/Music/f", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE, 0x3)          = 0
utimensat(4, NULL, {{1600461257, 619450219}, {1600461257, 619450219}}, 0) = 0
flistxattr(3, NULL, 0)                  = 24
flistxattr(3, "system.posix_acl_access\0", 24) = 24
open("/etc/xattr.conf", O_RDONLY)       = -1 ENOENT (No such file or directory)
fgetxattr(3, "system.posix_acl_access", 0x0, 0) = 68
fgetxattr(3, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68) = 68
fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68, 0) = 0
fgetxattr(3, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 132) = 68
fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68, 0) = 0
close(4)                                = 0
close(3)                                = 0
lstat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
newfstatat(AT_FDCWD, "f", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
unlinkat(AT_FDCWD, "f", 0)              = 0
lseek(0, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
close(0)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

I'm waaaay out of my depth here, but I'm guessing these lines are the important ones:

rename("f", "/data/Music/f")            = -1 EXDEV (Invalid cross-device link)
[... followed shortly by ...]
ioctl(4, BTRFS_IOC_CLONE, 0x3)          = 0

It looks like the rename call fails, and mv falls back to using a BTRFS-specific alternative.

[ianlancetaylor]

Interesting. I was not aware of that. I'll retitle this issue.

[codeaholics]

I've tried to follow the source for mv, and... well... I'm not a C programmer! 🤣 But ultimately, it ends up here: https://github.com/coreutils/coreutils/blob/master/src/copy.c#L405-L418, which, AFAICT, comes from here: https://github.com/coreutils/coreutils/blob/master/src/copy.c#L1459.

Ultimately, it appears to all be controlled by the defaulting of the following two flags in mv.c:
https://github.com/coreutils/coreutils/blob/master/src/mv.c#L113
https://github.com/coreutils/coreutils/blob/master/src/mv.c#L130

[codeaholics]

It's interesting that BTRFS reports those two directories as being on different filesystems, but is then happy to do a CoW!

[thanm]

Maybe related: https://superuser.com/questions/948808/why-is-moving-files-between-btrfs-subvolumes-an-expensive-operation