cmd/link: cannot fallocate file in tmpfs on EulerOS when cpuprofile is on

[alexanius]

Go version

go version devel go1.24-6edc1c23ed Thu Aug 22 01:18:23 2024 +0000 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/amarkin/.cache/go-build'
GOENV='/home/amarkin/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/amarkin/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/amarkin/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/amarkin/repo/go_bb'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/amarkin/repo/go_bb/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.24-6edc1c23ed Thu Aug 22 01:18:23 2024 +0000'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/amarkin/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/amarkin/repo/benchmarks/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2988943068=/tmp/go-build -gno-record-gcc-switches'

What did you do?

$ cat t.go
package main

import (
        "os"
        "runtime/pprof"
        "syscall"
)

func main() {
        f, _ := os.Create("t.pprof")
        pprof.StartCPUProfile(f)

        ff, _ := os.Create("/tmp/ttt")
        for i := 0 ; ; i++ {
                if  err := syscall.Fallocate(int(ff.Fd()), 0, 0, 250000000); err != syscall.EINTR {
                        break
                }
                println("Fail:", i)
        }
        println("Success")

        pprof.StopCPUProfile()
        f.Close()
}

$ ~/repo/CoreGo/bin/go run t.go
Fail: 0
Fail: 1
Fail: 2
Fail: 3
Fail: 4
Fail: 5
Fail: 6
...

What did you see happen?

The program can not make Fallocate and goes into infinite loop.

What did you expect to see?

The program makes changes the file size.

Explanation

This situation happens when few factors are met in the same time:

• EulerOs
• The cpu profiling is turned on
• We try to change size of the file on tmpfs

The real-world situation happened, when I tried to measure sweet benchmark in the pgo mode on my performance server. The problematic callstack is the following:

#0  syscall.Fallocate (fd=7, mode=0, off=0, len=241786748, err=...) at /home/amarkin/repo/go_bb/src/syscall/zsyscall_linux_amd64.go:387
#1  0x00000000005fe9da in cmd/link/internal/ld.(*OutBuf).fallocate (out=<optimized out>, size=<optimized out>, ~r0=...) at /home/amarkin/repo/go_bb/src/cmd/link/internal/ld/outbuf_linux.go:10
#2  0x00000000005fea8c in cmd/link/internal/ld.(*OutBuf).Mmap (out=0xc0000920e0, filesize=241786748, err=...) at /home/amarkin/repo/go_bb/src/cmd/link/internal/ld/outbuf_mmap.go:23
#3  0x00000000005fd075 in cmd/link/internal/ld.Main (arch=0x90e340, theArch=...) at /home/amarkin/repo/go_bb/src/cmd/link/internal/ld/main.go:444
#4  0x0000000000684cfb in main.main () at /home/amarkin/repo/go_bb/src/cmd/link/main.go:72

The problematic code is in the cmd/link/internal/ld.(*OutBuf).Mmap:

        for {
                if err = out.fallocate(filesize); err != syscall.EINTR {
                        break
                }
        }

The fallocate always returns EINTR and the code do not leave the loop. I found, that other people faced the similar problems, but their solution was the loop above.

My solution of this problem was to replace the fallocate with ftruncate syscall in the following way:

        for i := 0 ; ; i++ {
                if i < 100 {
                        if  err = out.fallocate(filesize); err != syscall.EINTR {
                                break
                        }
                } else {
                        if  err = out.ftruncate(filesize); err != syscall.EINTR {
                                break
                        }
                }
        }

That workaround worked, and I'd like to ask the advice what is the best way to solve the problem?

[gabyhelp]

Related Issues and Documentation

• debug/pe: fatal error: runtime: out of memory on NewFile #43827
• runtime: simple programs crash on linux/386 with go1.21 when build with -gcflags='all=-N -l' #61975 (closed)
• cmd/compile: seemingly stuck writing telemetry #68311 (closed)
• runtime: some kind deadlock issue with the new scheduler #4998 (closed)
• cmd/link: TestFallocate fallocate_test error #41127 (closed)
• cmd/link/internal/ld: fallocate_test.go failed test #50042 (closed)
• runtime: epollwait on fd 1 failed with 22 #68333 (closed)
• runtime: fatal error: checkdead: runnable g #40368 (closed)
• cmd/link: fallocate_test.go:46: unexpected disk usage: got 1 blocks, want 2048 #39905 (closed)
• runtime: frame pointer unwinding can fail on system goroutines #63630

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

Retrying on EINTR is the right fix.

[ianlancetaylor]

Hmmm, I see though, you are saying that the system call never succeeds. It always gets interrupted. That seems like it might be a kernel bug. I'm not familiar with EulerOS. Does this happen on other Linux distributions?

[alexanius]

Hmmm, I see though, you are saying that the system call never succeeds. It always gets interrupted. That seems like it might be a kernel bug. I'm not familiar with EulerOS. Does this happen on other Linux distributions?

As I can see - some users faced this problem (see link ). The kernel has the patch for this situation, but it is quite recent. I'll try to communicate with Euler developers and ask what they think.