os: cmd/go gets error "copy_file_range: function not implemented"

[m1d1]

Please file a new issue at golang.org/issue/new using this template:

What version of Go are you using (go version)?

Go on a VSERVER.

$ go version
go version go1.20.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.20"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.20/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build308414333=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.20.1 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.20.1
uname -sr: Linux 5.4.0
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.31-0ubuntu9.9) stable release version 2.31.

What did you do?

Tried to build or run a "hello world" Println example.

What did you expect to see?

"Hello World"

What did you see instead?

runtime: copying /usr/lib/go-1.20/src/runtime/asm_amd64.h to /tmp/go-build3218382935/b008/asm_GOARCH.h: write /tmp/go-build3218382935/b008/asm_GOARCH.h: copy_file_range: function not implement

Note

go 1.19 worked on the Vserver. go >= 1.20 doesn't.

[ianlancetaylor]

Are you sure that the error message is "function not implement" ? I don't see where that could come from. Could it be "function not implemented"?

What Linux kernel version are you running?

[m1d1]

Yes, it's "implemented" might have cut the 'ed' on accident.

┌─[root]
└─▪ go run main.go
runtime: copying /usr/lib/go-1.20/src/runtime/asm_amd64.h to /tmp/go-build682241010/b008/asm_GOARCH.h: write /tmp/go-build682241010/b008/asm_GOARCH.h: copy_file_range: function not implemented

uname -sr: Linux 5.4.0

[remover]

same issue. reverting to go 1.19.4 "fixes it"

[ccmtaylor]

I'm able to reproduce this, it seems to be a regression introduced in Go 1.20.0. To expand on @remover's comment: the exact message in our case is:

runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4200235085/b002/tls_GOARCH.h: write /tmp/go-build4200235085/b002/tls_GOARCH.h: copy_file_range: function not implemented

In our case, we see the error when running go run in an arm64 docker container on a amd64 debian bullseye host. The following steps reproduce it for me:

• create an amd64 Debian Bullseye VM in a cloud provider of your choice (we've seen this in both EC2 and GCP)
• install necessary packages for running arm64 docker images: sudo apt install libc6-arm64-cross qemu-user-static docker.io
• create an empty Go executable:

  package main
  func main() {}

• run it with sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.1 go run test.go

This works with Go 1.19.4, but fails with Go 1.20.0 and 1.20.1:

$ sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.1 go run test.go
  runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4200235085/b002/tls_GOARCH.h: write /tmp/go-build4200235085/b002/tls_GOARCH.h: copy_file_range: function not implemented

$ sudo docker run --rm -ti --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.20.0 go run test.go
runtime: copying /usr/local/go/src/runtime/tls_arm64.h to /tmp/go-build4014913792/b002/tls_GOARCH.h: write /tmp/go-build4014913792/b002/tls_GOARCH.h: copy_file_range: function not implemented

$ sudo docker run --rm --platform linux/arm64/v8 -v $(pwd)/test.go:/go/test.go golang:1.19.4 go run test.go
<no output>

[ccmtaylor]

I was able to remove docker from the picture. Running directly with qemu results in the same behaviour:

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ arm64/go/bin/go version
go version go1.20.1 linux/arm64

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ arm64/go/bin/go run test.go 
runtime: copying /home/christopher_taylor/arm64/go/src/runtime/tls_arm64.h to /tmp/go-build1029963439/b002/tls_GOARCH.h: write /tmp/go-build1029963439/b002/tls_GOARCH.h: copy_file_range: function not implemented

I think this is interesting because similar to @m1d1, there's a virtualization layer involved. I don't know if VSERVER is qemu-based?

For completeness, to see if this is something to do with qemu, I tried running the same in qemu-x86_64-static, but only got segfaults, so I didn't look further.

$ amd64/go/bin/go version
go version go1.20.1 linux/amd64

$ qemu-x86_64-static amd64/go/bin/go version
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

[m1d1]

@ccmtaylor:

┌─[root]
└─▪ virt-what 
openvz
lxc

[ianlancetaylor]

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

[gopherbot]

Change https://go.dev/cl/470016 mentions this issue: Revert "internal/poll: drop redundant ENOSYS in CopyFileRange"

[ianlancetaylor]

@gopherbot Please open a backport to 1.20.

On some systems copy_file_range appears to fail with ENOSYS, even though it seems that it shouldn't. On those systems the go tool doesn't work at all.

[gopherbot]

Backport issue(s) opened: #58627 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[panjf2000]

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

Also, could you guys try to run the example C code of copy_file_range in man page to see if your OS with kernel version > 5.3 surely get ENOSYS when calling copy_file_range? If so, I believe we should also file a bug report for Linux kernel?

[ccmtaylor]

@ianlancetaylor @panjf2000 I'll give those a try today

[ccmtaylor]

I can confirm that https://go.dev/cl/470016 fixes the issue for us. I cherry-picked the CL onto the go1.20.1 tag in a fresh golang/go checkout, buit it with CGO_ENABLED=0 GOROOT_BOOTSTRAP=~/amd64/go GOARCH=arm64 ./make.bash and ran these to confirm:

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ goroot/bin/linux_arm64/go version
go version go1.20.1 linux/arm64
$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ goroot/bin/linux_arm64/go run test.go 

(CGO_ENABLED=0 was because I didn't have a cross-compiler installed. Trying again now) I rebuilt Go without CGO_ENABLED=0 and re-ran the tests successfully. Sounds like we're good to go.

[ccmtaylor]

Can people who are having this problem please see if https://go.dev/cl/470016 fixes it. Thanks.

Also, could you guys try to run the example C code of copy_file_range in man page to see if your OS with kernel version > 5.3 surely get ENOSYS when calling copy_file_range? If so, I believe we should also file a bug report for Linux kernel?

@panjf2000 I just gave this a try, too. Looks like copy_file_range doesn't work in this setup (Debian Bullseye amd64, running arm64 binaries via qemu):

$ cat [sample code from https://man7.org/linux/man-pages/man2/copy_file_range.2.html#EXAMPLES] > copy_file_range_test.c

$  CC=aarch64-linux-gnu-gcc make copy_file_range_test
aarch64-linux-gnu-gcc     copy_file_range_test.c   -o copy_file_range_test

$ touch foo

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu ./copy_file_range_test foo bar
copy_file_range: Function not implemented

[ccmtaylor]

I see that the copy_file_range implementation in qemu may return ENOSYS under some circumstances, so maybe that's what we're seeing?

[panjf2000]

@ccmtaylor Thank you for the effort, we now know the behavior of qemu, it does look a bit unusual.

On the other hand, I'm also interested in how the real machine (other than emulator and virtualizer) acts on copy_file_range, could you also run the C example code on your machine and observe what happens, thanks! @m1d1

[ccmtaylor]

sure! the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1):

$ uname -a
Linux ccmtaylor-bullseye-test 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

$ CC=gcc make copy_file_range_test
gcc     copy_file_range_test.c   -o copy_file_range_test

$ ./copy_file_range_test foo bar

$ ls -1s foo bar
0 bar
0 foo

[panjf2000]

the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1)

What about go1.20? does it also work with copy_file_range?

[ccmtaylor]

the copy_file_range sample code works fine running natively on the host (an e2-medium instance in europe-west1)

What about go1.20? does it also work with copy_file_range?

not sure what you're asking. Yes, the release binaries for Go 1.20.1 work fine on the host:

# release binaries
$ amd64/go/bin/go run test.go
Hello World

# go1.20.1 patched with https://go.dev/cl/470016
$ goroot/bin/go run test.go
Hello World

[ianlancetaylor]

@ccmtaylor Thanks.

[gopherbot]

Change https://go.dev/cl/470316 mentions this issue: [release-branch.go1.20] Revert "internal/poll: drop redundant ENOSYS in CopyFileRange"