Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: if rename system call fails, try ioctl(FICLONE) #41487

Open
codeaholics opened this issue Sep 18, 2020 · 6 comments
Open

os: if rename system call fails, try ioctl(FICLONE) #41487

codeaholics opened this issue Sep 18, 2020 · 6 comments

Comments

@codeaholics
Copy link

@codeaholics codeaholics commented Sep 18, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/danny/Library/Caches/go-build"
GOENV="/Users/danny/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/danny/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/danny/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/danny/workspace/media-sort/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/36/kj75t85j1m57cg047cr5dh4m0000gn/T/go-build628346493=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I am cross-compiling on Mac for Linux

Linux execution env: Linux nas 4.4.190.x86_64.1 #1 SMP Mon Oct 28 01:55:46 UTC 2019 x86_64 GNU/Linux

Build command: GOOS=linux GOARCH=amd64 go build -v

func move(src, dst string) error {
	err := os.Rename(src, dst)
	// cross device move
	if err != nil && strings.Contains(err.Error(), "cross-device") {
		if err := copy(src, dst); err != nil {
			return err
		}
		if err := os.Remove(src); err != nil {
			return err
		}
	}
	return nil
}

What did you expect to see?

os.Rename() to succeed because the source and target are on the same filesystem.

What did you see instead?

os.Rename() returns "invalid cross-device link" and the above code falls back to copy-and-delete. If I do an mv on the command line, it completes immediately - even for multi-gigabyte files.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 18, 2020

os.Rename is a fairly thin wrapper around syscall.Rename, which just calls the rename system call. If possible, please run the final GNU/Linux executable under strace -f and see what the rename system call returns.

@ianlancetaylor ianlancetaylor changed the title os.Rename() returns "invalid cross-device link" even though source and dest are on same device os: Rename returns "invalid cross-device link" even though source and dest are on same device Sep 18, 2020
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Sep 18, 2020
@codeaholics
Copy link
Author

@codeaholics codeaholics commented Sep 18, 2020

Thanks for your quick response @ianlancetaylor. To test, I made an empty file, f, in /data/Videos on my NAS, and mv-ed it to /data/Music. Here's the strace:

execve("/bin/mv", ["mv", "f", "/data/Music"], [/* 17 vars */]) = 0
brk(0)                                  = 0x16eb000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d8000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=28370, ...}) = 0
mmap(NULL, 28370, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe1406d1000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20c\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=142728, ...}) = 0
mmap(NULL, 2246896, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe140295000
mprotect(0x7fe1402b6000, 2097152, PROT_NONE) = 0
mmap(0x7fe1404b6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7fe1404b6000
mmap(0x7fe1404b8000, 6384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe1404b8000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\37\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=35288, ...}) = 0
mmap(NULL, 2130592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe14008c000
mprotect(0x7fe140094000, 2093056, PROT_NONE) = 0
mmap(0x7fe140293000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fe140293000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18640, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d0000
mmap(NULL, 2113912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13fe87000
mprotect(0x7fe13fe8b000, 2093056, PROT_NONE) = 0
mmap(0x7fe14008a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fe14008a000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1738176, ...}) = 0
mmap(NULL, 3844640, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13fadc000
mprotect(0x7fe13fc7d000, 2097152, PROT_NONE) = 0
mmap(0x7fe13fe7d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a1000) = 0x7fe13fe7d000
mmap(0x7fe13fe83000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe13fe83000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=448440, ...}) = 0
mmap(NULL, 2543976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f86e000
mprotect(0x7fe13f8da000, 2097152, PROT_NONE) = 0
mmap(0x7fe13fada000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c000) = 0x7fe13fada000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406cf000
mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f66a000
mprotect(0x7fe13f66d000, 2093056, PROT_NONE) = 0
mmap(0x7fe13f86c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fe13f86c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320n\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137384, ...}) = 0
mmap(NULL, 2213008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe13f44d000
mprotect(0x7fe13f465000, 2093056, PROT_NONE) = 0
mmap(0x7fe13f664000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fe13f664000
mmap(0x7fe13f666000, 13456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe13f666000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406ce000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406cc000
arch_prctl(ARCH_SET_FS, 0x7fe1406cc800) = 0
mprotect(0x7fe13fe7d000, 16384, PROT_READ) = 0
mprotect(0x7fe13f664000, 4096, PROT_READ) = 0
mprotect(0x7fe13f86c000, 4096, PROT_READ) = 0
mprotect(0x7fe13fada000, 4096, PROT_READ) = 0
mprotect(0x7fe14008a000, 4096, PROT_READ) = 0
mprotect(0x7fe140293000, 4096, PROT_READ) = 0
mprotect(0x7fe1404b6000, 4096, PROT_READ) = 0
mprotect(0x61e000, 4096, PROT_READ)     = 0
mprotect(0x7fe1406da000, 4096, PROT_READ) = 0
munmap(0x7fe1406d1000, 28370)           = 0
set_tid_address(0x7fe1406ccad0)         = 23496
set_robust_list(0x7fe1406ccae0, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7fe13f4539b0, [], SA_RESTORER|SA_SIGINFO, 0x7fe13f45c890}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fe13f453a40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fe13f45c890}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffc57d353f0) = -1 ENOENT (No such file or directory)
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=965552, f_bfree=722015, f_bavail=665542, f_files=1048576, f_ffree=1032627, f_fsid={-1827525020, -625743731}, f_namelen=255, f_frsize=4096}) = 0
brk(0)                                  = 0x16eb000
brk(0x170c000)                          = 0x170c000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1406d7000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tt"..., 1024) = 370
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fe1406d7000, 4096)            = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=6839968, ...}) = 0
mmap(NULL, 6839968, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe13edc7000
close(3)                                = 0
geteuid()                               = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("/data/Music", {st_mode=S_IFDIR|0777, st_size=38, ...}) = 0
lstat("f", {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
lstat("/data/Music/f", 0x7ffc57d35060)  = -1 ENOENT (No such file or directory)
rename("f", "/data/Music/f")            = -1 EXDEV (Invalid cross-device link)
unlink("/data/Music/f")                 = -1 ENOENT (No such file or directory)
open("f", O_RDONLY|O_NOFOLLOW)          = 3
fstat(3, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
open("/data/Music/f", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE, 0x3)          = 0
utimensat(4, NULL, {{1600461257, 619450219}, {1600461257, 619450219}}, 0) = 0
flistxattr(3, NULL, 0)                  = 24
flistxattr(3, "system.posix_acl_access\0", 24) = 24
open("/etc/xattr.conf", O_RDONLY)       = -1 ENOENT (No such file or directory)
fgetxattr(3, "system.posix_acl_access", 0x0, 0) = 68
fgetxattr(3, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68) = 68
fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68, 0) = 0
fgetxattr(3, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 132) = 68
fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x02\x00\x07\x00b\x00\x00\x00\x02\x00\x07\x00c\x00\x00\x00\x04\x00\x07\x00\xff\xff\xff\xff\x08\x00\x07\x00b\x00\x00\x00\x08\x00\x07\x00c\x00\x00\x00\x10\x00\x06\x00\xff\xff\xff\xff \x00\x06\x00\xff\xff\xff\xff", 68, 0) = 0
close(4)                                = 0
close(3)                                = 0
lstat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
newfstatat(AT_FDCWD, "f", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
unlinkat(AT_FDCWD, "f", 0)              = 0
lseek(0, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
close(0)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

I'm waaaay out of my depth here, but I'm guessing these lines are the important ones:

rename("f", "/data/Music/f")            = -1 EXDEV (Invalid cross-device link)
[... followed shortly by ...]
ioctl(4, BTRFS_IOC_CLONE, 0x3)          = 0

It looks like the rename call fails, and mv falls back to using a BTRFS-specific alternative.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 18, 2020

Interesting. I was not aware of that. I'll retitle this issue.

@ianlancetaylor ianlancetaylor changed the title os: Rename returns "invalid cross-device link" even though source and dest are on same device os: if rename system call fails, try ioctl(FICLONE) Sep 18, 2020
@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Unplanned Sep 18, 2020
@codeaholics
Copy link
Author

@codeaholics codeaholics commented Sep 18, 2020

I've tried to follow the source for mv, and... well... I'm not a C programmer! 🤣 But ultimately, it ends up here: https://github.com/coreutils/coreutils/blob/master/src/copy.c#L405-L418, which, AFAICT, comes from here: https://github.com/coreutils/coreutils/blob/master/src/copy.c#L1459.

Ultimately, it appears to all be controlled by the defaulting of the following two flags in mv.c:
https://github.com/coreutils/coreutils/blob/master/src/mv.c#L113
https://github.com/coreutils/coreutils/blob/master/src/mv.c#L130

@codeaholics
Copy link
Author

@codeaholics codeaholics commented Sep 18, 2020

It's interesting that BTRFS reports those two directories as being on different filesystems, but is then happy to do a CoW!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.