Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: cannot open file ... cannot allocate memory #52716

Open
MikeSpreitzer opened this issue May 4, 2022 · 5 comments
Open

cmd/link: cannot open file ... cannot allocate memory #52716

MikeSpreitzer opened this issue May 4, 2022 · 5 comments
Labels
WaitingForInfo

Comments

@MikeSpreitzer
Copy link

@MikeSpreitzer MikeSpreitzer commented May 4, 2022

What version of Go are you using (go version)?

# go version
go version go1.18.1 linux/amd64

Does this issue reproduce with the latest release?

I am using the latest release.

What operating system and processor architecture are you using (go env)?

go env Output
# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mspreitz/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mspreitz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mspreitz/go/src/k8s.io/kubernetes/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1360746828=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to use hack/local-up-cluster.sh of Kubernetes release-1.24 in a freshly constructed VM that has plenty of memory, but everything I try fails with the linker complaining that it can not open a file and can not allocate memory.

I monitored with htop in a separate window, the memory usage never got above about 25%.

Here is some more evidence in typescript form.

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git rev-parse HEAD
4e82620fad0b6d57b2e980dcc7f4f537d812b82d

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git status
On branch release-1.24
Your branch is up to date with 'upstream/release-1.24'.

I ran the local-up-cluster script as root, after first removing the _output and /tmp files/directories owned by my ordinary user. Following is the latest trial.

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# LOG_LEVEL=5 hack/local-up-cluster.sh 
make: Entering directory '/home/mspreitz/go/src/k8s.io/kubernetes'
make[1]: Entering directory '/home/mspreitz/go/src/k8s.io/kubernetes'
+++ [0504 22:54:52] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
make[1]: Leaving directory '/home/mspreitz/go/src/k8s.io/kubernetes'
+++ [0504 22:55:14] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-scheduler
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/5b/5bd65f227d3f0f4a83ff704b25f53cd2e580b4e8084c6b15bbf2d6cc40ecb0ee-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/5b/5bd65f227d3f0f4a83ff704b25f53cd2e580b4e8084c6b15bbf2d6cc40ecb0ee-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/52/528d8562ae000b7ab4c219cfe549668f86c7ea1fdeee12843cba0c1a53c15e59-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/52/528d8562ae000b7ab4c219cfe549668f86c7ea1fdeee12843cba0c1a53c15e59-d: cannot allocate memory
!!! [0504 22:55:29] Call tree:
!!! [0504 22:55:29]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0504 22:55:29]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0504 22:55:29]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0504 22:55:29] Call tree:
!!! [0504 22:55:29]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0504 22:55:29] Call tree:
!!! [0504 22:55:29]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1
make: Leaving directory '/home/mspreitz/go/src/k8s.io/kubernetes'

Here is more information about the VM.

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ head -5 /proc/meminfo 
MemTotal:       25822384 kB
MemFree:        23363660 kB
MemAvailable:   25146860 kB
Buffers:           28952 kB
Cached:          2033048 kB

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ tail -30 /proc/cpuinfo | head -10
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 158
model name	: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
stepping	: 13
cpu MHz		: 2303.998

And, in the shell I used to run the local-up-cluster script:

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -l _output/local/go/cache/5b/5bd65f227d3f0f4a83ff704b25f53cd2e580b4e8084c6b15bbf2d6cc40ecb0ee-d
-rw-r--r-- 1 mspreitz mspreitz 280754 May  4 21:59 _output/local/go/cache/5b/5bd65f227d3f0f4a83ff704b25f53cd2e580b4e8084c6b15bbf2d6cc40ecb0ee-d

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -l _output/local/go/cache/52/528d8562ae000b7ab4c219cfe549668f86c7ea1fdeee12843cba0c1a53c15e59-d
-rw-r--r-- 1 mspreitz mspreitz 83680 May  4 22:04 _output/local/go/cache/52/528d8562ae000b7ab4c219cfe549668f86c7ea1fdeee12843cba0c1a53c15e59-d

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ulimit -S -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 100429
max locked memory           (kbytes, -l) 3227796
max memory size             (kbytes, -m) unlimited
open files                          (-n) 102400
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 100429
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ulimit -H -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 100429
max locked memory           (kbytes, -l) 3227796
max memory size             (kbytes, -m) unlimited
open files                          (-n) 102400
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 100429
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# printenv | grep GO
GOPATH=/home/mspreitz/go

What did you expect to see?

No complaints about memory from inside a VM with plenty of memory.

What did you see instead?

See the error messages about memory above.

@dr2chase
Copy link
Contributor

@dr2chase dr2chase commented May 4, 2022

Assume I know nothing about kubernetes and have never installed it. Can you provide me with a shell script, end-to-end, that would reproduce this? Partly this is because I'm skeptical it's a linker bug, and partly it's because I don't know much about k8s and my failure to reproduce would not be definitive information.

@dr2chase dr2chase added the WaitingForInfo label May 4, 2022
@MikeSpreitzer
Copy link
Author

@MikeSpreitzer MikeSpreitzer commented May 5, 2022

Oh, I should add this bit of evidence too:

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              2.5G  1.1M  2.5G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   96G  8.9G   83G  10% /
tmpfs                               13G     0   13G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
go                                 932G  604G  328G  65% /home/mspreitz/go
/dev/sda2                          2.0G  126M  1.7G   7% /boot
tmpfs                              2.5G  4.0K  2.5G   1% /run/user/0

I will see if I can narrow down the scenario.

@MikeSpreitzer MikeSpreitzer changed the title affected/package: go/pkg/tool/linux_amd64/link: cannot open file ... cannot allocate memory May 5, 2022
@MikeSpreitzer
Copy link
Author

@MikeSpreitzer MikeSpreitzer commented May 5, 2022

Here is a slightly simpler way to produce the problem. It is odd. First, do a successful build. Then try building again, and the problem happens! Following is a typescript.

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# make clean
+++ [0505 03:14:39] Verifying Prerequisites....
+++ [0505 03:14:39] Removing _output directory
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 03:15:01] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 03:15:33] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen (non-static)
+++ [0505 03:15:38] Generating prerelease lifecycle code for 26 targets
+++ [0505 03:15:41] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/deepcopy-gen (non-static)
+++ [0505 03:15:43] Generating deepcopy code for 236 targets
+++ [0505 03:15:51] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen (non-static)
+++ [0505 03:15:53] Generating defaulter code for 92 targets
+++ [0505 03:16:02] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/conversion-gen (non-static)
+++ [0505 03:16:04] Generating conversion code for 129 targets
+++ [0505 03:16:22] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/kube-openapi/cmd/openapi-gen (non-static)
+++ [0505 03:16:30] Generating openapi code for KUBE
+++ [0505 03:16:45] Generating openapi code for AGGREGATOR
+++ [0505 03:16:46] Generating openapi code for APIEXTENSIONS
+++ [0505 03:16:48] Generating openapi code for CODEGEN
+++ [0505 03:16:49] Generating openapi code for SAMPLEAPISERVER
+++ [0505 03:16:51] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 03:26:00] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 03:26:21] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/cd/cd709073b732afa17dff0b75b98ace5b805aa01e944bbc23d619b72720f3ac50-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/cd/cd709073b732afa17dff0b75b98ace5b805aa01e944bbc23d619b72720f3ac50-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-apiserver
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/71/71cc244fe95bb3a2f05ff7e0317cf4f29469b8fce6af9bc16e9c6814b458f94e-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/71/71cc244fe95bb3a2f05ff7e0317cf4f29469b8fce6af9bc16e9c6814b458f94e-d: cannot allocate memory
!!! [0505 03:26:36] Call tree:
!!! [0505 03:26:36]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 03:26:36]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 03:26:36]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 03:26:37] Call tree:
!!! [0505 03:26:37]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 03:26:37] Call tree:
!!! [0505 03:26:37]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1

@MikeSpreitzer
Copy link
Author

@MikeSpreitzer MikeSpreitzer commented May 5, 2022

By using ps in another shell, I can see that the main go command is

go install -installsuffix=static -gcflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes  -asmflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes "-ldflags=all=-X 'k8s.io/kubernetes/vendor/k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/kubernetes/vendor/k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z'  -s -w" -tags=selinux,notest, k8s.io/kubernetes/cmd/kubectl k8s.io/kubernetes/cmd/kube-apiserver k8s.io/kubernetes/cmd/kube-controller-manager k8s.io/kubernetes/cmd/kube-proxy k8s.io/kubernetes/cmd/kube-scheduler

Oddly, when I try that directly from the shell, I get a complaint about accessing the "VCS status". Indeed, git is reticent to touch its metadata in this situation, with git being run by a different user than the one that did the git clone and owns the directories and files.

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# go install -installsuffix=static -gcflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes  -asmflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes "-ldflags=all=-X 'k8s.io/kubernetes/vendor/k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/kubernetes/vendor/k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z'  -s -w" -tags=selinux,notest, k8s.io/kubernetes/cmd/kubectl k8s.io/kubernetes/cmd/kube-apiserver k8s.io/kubernetes/cmd/kube-controller-manager k8s.io/kubernetes/cmd/kube-proxy k8s.io/kubernetes/cmd/kube-scheduler
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.

If I take the hint, I can issue a go install command that usually works.

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# date; go install -buildvcs=false -installsuffix=static -gcflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes  -asmflags=all=-trimpath=/home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes "-ldflags=all=-X 'k8s.io/kubernetes/vendor/k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/kubernetes/vendor/k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/client-go/pkg/version.buildDate=2022-05-05T03:16:51Z' -X 'k8s.io/component-base/version.buildDate=2022-05-05T03:16:51Z'  -s -w" -tags=selinux,notest, k8s.io/kubernetes/cmd/kubectl k8s.io/kubernetes/cmd/kube-apiserver k8s.io/kubernetes/cmd/kube-controller-manager k8s.io/kubernetes/cmd/kube-proxy k8s.io/kubernetes/cmd/kube-scheduler
Thu May  5 04:24:32 AM UTC 2022
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 

I once saw the direct go install file with the same complaint as make, but have been unable to make it happen again.

@mengzhuo mengzhuo changed the title go/pkg/tool/linux_amd64/link: cannot open file ... cannot allocate memory cmd/link: cannot open file ... cannot allocate memory May 5, 2022
@MikeSpreitzer
Copy link
Author

@MikeSpreitzer MikeSpreitzer commented May 5, 2022

Hey, @dr2chase , thanks for responding. My latest theory is that the fault lies not with go, nor even Kubernetes, but with the VirtualBox mechanism I am using to mount part of the host filesystem into the guest. The troubles I reported here are from inside a VM I just made using VirtualBox. I posted on a VBox forum at https://forums.virtualbox.org/viewtopic.php?f=3&t=105969

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WaitingForInfo
Projects
None yet
Development

No branches or pull requests

2 participants