Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make fails on second and subsequent runs, with "...go/pkg/tool/linux_amd64/link: cannot open file ... cannot allocate memory" #109808

Closed
MikeSpreitzer opened this issue May 5, 2022 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/usability Categorizes an issue or PR as relevant to SIG Usability.

Comments

@MikeSpreitzer
Copy link
Member

MikeSpreitzer commented May 5, 2022

What happened?

I made myself a shiny new Ubuntu 22.04 VM and tricked it out for building Kubernetes. But hack/local-up-cluster.sh repeatedly failed with a bogus complaint from the linker about not being able to open some files due to "cannot allocate memory". This VM has 24 GiB of memory and 4 vCPUs. I monitored CPU and memory usage in another window with htop; the yellow part of the memory bar never got near the right end.

Looking through that scripting, I was able to narrow the problem down to a make command that works the first time and fails on subsequent runs.

I did the git clone as my ordinary user. I checked out the latest release branch. Here is the state:

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git rev-parse HEAD
4e82620fad0b6d57b2e980dcc7f4f537d812b82d
mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git status
On branch release-1.24
Your branch is up to date with 'upstream/release-1.24'.

I tried hack/local-up-cluster.sh as root, and thus also tried building as root. Following is a demonstration of the problem (another make clean was done just before this typescript).

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# make clean
+++ [0505 04:53:11] Verifying Prerequisites....

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -ld _output
ls: cannot access '_output': No such file or directory

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# date; make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
Thu May  5 04:53:29 AM UTC 2022
+++ [0505 04:53:34] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 04:54:08] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen (non-static)
+++ [0505 04:54:13] Generating prerelease lifecycle code for 26 targets
+++ [0505 04:54:16] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/deepcopy-gen (non-static)
+++ [0505 04:54:18] Generating deepcopy code for 236 targets
+++ [0505 04:54:27] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen (non-static)
+++ [0505 04:54:29] Generating defaulter code for 92 targets
+++ [0505 04:54:38] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/conversion-gen (non-static)
+++ [0505 04:54:40] Generating conversion code for 129 targets
+++ [0505 04:54:58] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/kube-openapi/cmd/openapi-gen (non-static)
+++ [0505 04:55:05] Generating openapi code for KUBE
+++ [0505 04:55:21] Generating openapi code for AGGREGATOR
+++ [0505 04:55:22] Generating openapi code for APIEXTENSIONS
+++ [0505 04:55:24] Generating openapi code for CODEGEN
+++ [0505 04:55:25] Generating openapi code for SAMPLEAPISERVER
+++ [0505 04:55:27] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -l _output/bin/*
-rwxr-xr-x 1 mspreitz mspreitz 102326272 May  5 05:00 _output/bin/cloud-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz   6361088 May  5 04:54 _output/bin/conversion-gen
-rwxr-xr-x 1 mspreitz mspreitz   6090752 May  5 04:54 _output/bin/deepcopy-gen
-rwxr-xr-x 1 mspreitz mspreitz   6094848 May  5 04:54 _output/bin/defaulter-gen
-rwxr-xr-x 1 mspreitz mspreitz   2310144 May  5 04:53 _output/bin/go2make
-rwxr-xr-x 1 mspreitz mspreitz 125857792 May  5 05:00 _output/bin/kube-apiserver
-rwxr-xr-x 1 mspreitz mspreitz 115503104 May  5 05:00 _output/bin/kube-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz  45699072 May  5 05:00 _output/bin/kubectl
-rwxr-xr-x 1 mspreitz mspreitz 116339688 May  5 05:00 _output/bin/kubelet
-rwxr-xr-x 1 mspreitz mspreitz  41754624 May  5 05:00 _output/bin/kube-proxy
-rwxr-xr-x 1 mspreitz mspreitz  47140864 May  5 05:00 _output/bin/kube-scheduler
-rwxr-xr-x 1 mspreitz mspreitz   8388608 May  5 04:55 _output/bin/openapi-gen
-rwxr-xr-x 1 mspreitz mspreitz   6066176 May  5 04:54 _output/bin/prerelease-lifecycle-gen

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# date; make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
Thu May  5 05:01:02 AM UTC 2022
+++ [0505 05:01:07] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:01:28] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/27/27210e7f6f4b061bd26fb6e54f314e32ab50605b4ee52fad3f78d1988c5f3622-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/27/27210e7f6f4b061bd26fb6e54f314e32ab50605b4ee52fad3f78d1988c5f3622-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-apiserver
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/52/5251e854798cc399c901696a22cb98ccd4483ebdd8b30ea481b5cc1c35a6350c-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/52/5251e854798cc399c901696a22cb98ccd4483ebdd8b30ea481b5cc1c35a6350c-d: cannot allocate memory
!!! [0505 05:01:44] Call tree:
!!! [0505 05:01:44]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 05:01:44]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 05:01:44]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:01:44] Call tree:
!!! [0505 05:01:44]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:01:44] Call tree:
!!! [0505 05:01:44]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -l _output/bin/*
-rwxr-xr-x 1 mspreitz mspreitz 102326272 May  5 05:00 _output/bin/cloud-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz   6361088 May  5 04:54 _output/bin/conversion-gen
-rwxr-xr-x 1 mspreitz mspreitz   6090752 May  5 04:54 _output/bin/deepcopy-gen
-rwxr-xr-x 1 mspreitz mspreitz   6094848 May  5 04:54 _output/bin/defaulter-gen
-rwxr-xr-x 1 mspreitz mspreitz   2310144 May  5 05:01 _output/bin/go2make
-rwxr-xr-x 1 mspreitz mspreitz 125857792 May  5 05:00 _output/bin/kube-apiserver
-rwxr-xr-x 1 mspreitz mspreitz 115503104 May  5 05:00 _output/bin/kube-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz  45699072 May  5 05:00 _output/bin/kubectl
-rwxr-xr-x 1 mspreitz mspreitz 116339688 May  5 05:00 _output/bin/kubelet
-rwxr-xr-x 1 mspreitz mspreitz  41754624 May  5 05:00 _output/bin/kube-proxy
-rwxr-xr-x 1 mspreitz mspreitz  47140864 May  5 05:00 _output/bin/kube-scheduler
-rwxr-xr-x 1 mspreitz mspreitz   8388608 May  5 04:55 _output/bin/openapi-gen
-rwxr-xr-x 1 mspreitz mspreitz   6066176 May  5 04:54 _output/bin/prerelease-lifecycle-gen

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# date; make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
Thu May  5 05:02:10 AM UTC 2022
+++ [0505 05:02:16] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:02:37] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/9d/9dce2e2e9be43d4769a428f2432f411401618f65bbfc048dbe223c19ee1d6f39-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/9d/9dce2e2e9be43d4769a428f2432f411401618f65bbfc048dbe223c19ee1d6f39-d: cannot allocate memory
!!! [0505 05:02:58] Call tree:
!!! [0505 05:02:58]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 05:02:58]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 05:02:58]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:02:58] Call tree:
!!! [0505 05:02:58]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:02:58] Call tree:
!!! [0505 05:02:58]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 
root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# 

I then proceeded to remove the target binaries and try the build again; it still failed.

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# rm _output/bin/kube* _output/bin/cloud-controller-manager 

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ls -l _output/bin/*
-rwxr-xr-x 1 mspreitz mspreitz 6361088 May  5 04:54 _output/bin/conversion-gen
-rwxr-xr-x 1 mspreitz mspreitz 6090752 May  5 04:54 _output/bin/deepcopy-gen
-rwxr-xr-x 1 mspreitz mspreitz 6094848 May  5 04:54 _output/bin/defaulter-gen
-rwxr-xr-x 1 mspreitz mspreitz 2310144 May  5 05:02 _output/bin/go2make
-rwxr-xr-x 1 mspreitz mspreitz 8388608 May  5 04:55 _output/bin/openapi-gen
-rwxr-xr-x 1 mspreitz mspreitz 6066176 May  5 04:54 _output/bin/prerelease-lifecycle-gen

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# date; make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
Thu May  5 05:08:26 AM UTC 2022
+++ [0505 05:08:32] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:08:57] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-scheduler
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/64/649aaabf77d20f164fa1dcb7639eed7897d433f759bec91f8c2daaad20a0175b-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/64/649aaabf77d20f164fa1dcb7639eed7897d433f759bec91f8c2daaad20a0175b-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/4a/4ac1cb17b89d15e81bbd9ab71fcb3817e0b24bccfa2cd60ef2b4ce81e70dc840-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/4a/4ac1cb17b89d15e81bbd9ab71fcb3817e0b24bccfa2cd60ef2b4ce81e70dc840-d: cannot allocate memory
!!! [0505 05:09:14] Call tree:
!!! [0505 05:09:14]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 05:09:14]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 05:09:14]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:09:14] Call tree:
!!! [0505 05:09:14]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:09:14] Call tree:
!!! [0505 05:09:14]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1

What did you expect to happen?

I expected no bogus complaints about memory.

How can we reproduce it (as minimally and precisely as possible)?

The evidence above is pretty direct.

Anything else we need to know?

From the shell running make:

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ulimit -S -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 100429
max locked memory           (kbytes, -l) 3227796
max memory size             (kbytes, -m) unlimited
open files                          (-n) 102400
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 100429
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# ulimit -H -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 100429
max locked memory           (kbytes, -l) 3227796
max memory size             (kbytes, -m) unlimited
open files                          (-n) 102400
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 100429
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Also:

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# go version
go version go1.18.1 linux/amd64

root@ubu22:/home/mspreitz/go/src/k8s.io/kubernetes# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              2.5G  1.3M  2.5G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   96G   11G   81G  13% /
tmpfs                               13G     0   13G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
go                                 932G  613G  320G  66% /home/mspreitz/go
/dev/sda2                          2.0G  126M  1.7G   7% /boot
tmpfs                              2.5G  4.0K  2.5G   1% /run/user/0
tmpfs                              2.5G  4.0K  2.5G   1% /run/user/1000

/home/mspreitz/go inside this VM is mounted from the host's filesystem.

Kubernetes version

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git rev-parse HEAD
4e82620fad0b6d57b2e980dcc7f4f537d812b82d
mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ git status
On branch release-1.24
Your branch is up to date with 'upstream/release-1.24'.

Cloud provider

VirtualBox 6.1.34 on my MaOS 12.3.1 laptop

OS version

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Install tools

None

Container runtime (CRI) and version (if applicable)

N/A

Related plugins (CNI, CSI, ...) and versions (if applicable)

N/A
@MikeSpreitzer MikeSpreitzer added the kind/bug Categorizes issue or PR as related to a bug. label May 5, 2022
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label May 5, 2022
@k8s-ci-robot
Copy link
Contributor

@MikeSpreitzer: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label May 5, 2022
@MikeSpreitzer
Copy link
Member Author

MikeSpreitzer commented May 5, 2022

/sig usability
That's a rough guess. Is there a better choice?
See also golang/go#52716

@k8s-ci-robot k8s-ci-robot added sig/usability Categorizes an issue or PR as relevant to SIG Usability. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 5, 2022
@MikeSpreitzer
Copy link
Member Author

BTW, I get the same failure when building as my regular user. Following is a typescript, picking up just after a make clean that I ran as root.

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 05:29:29] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:30:04] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen (non-static)
+++ [0505 05:30:09] Generating prerelease lifecycle code for 26 targets
+++ [0505 05:30:14] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/deepcopy-gen (non-static)
+++ [0505 05:30:17] Generating deepcopy code for 236 targets
+++ [0505 05:30:28] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen (non-static)
+++ [0505 05:30:29] Generating defaulter code for 92 targets
+++ [0505 05:30:42] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/conversion-gen (non-static)
+++ [0505 05:30:44] Generating conversion code for 129 targets
+++ [0505 05:31:04] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/kube-openapi/cmd/openapi-gen (non-static)
+++ [0505 05:31:11] Generating openapi code for KUBE
+++ [0505 05:31:26] Generating openapi code for AGGREGATOR
+++ [0505 05:31:28] Generating openapi code for APIEXTENSIONS
+++ [0505 05:31:30] Generating openapi code for CODEGEN
+++ [0505 05:31:31] Generating openapi code for SAMPLEAPISERVER
+++ [0505 05:31:35] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 05:43:10] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:43:33] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/53/53e62a2c03132bf9b805260b78dc158c359f04215ebbe27d148080b0d77958fc-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/53/53e62a2c03132bf9b805260b78dc158c359f04215ebbe27d148080b0d77958fc-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-apiserver
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/36/3651855f60aa07db33d318babe73a621d3063ab56c9004976a6162acafbeec19-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/36/3651855f60aa07db33d318babe73a621d3063ab56c9004976a6162acafbeec19-d: cannot allocate memory
!!! [0505 05:43:53] Call tree:
!!! [0505 05:43:53]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 05:43:53]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 05:43:53]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:43:54] Call tree:
!!! [0505 05:43:54]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 05:43:54] Call tree:
!!! [0505 05:43:54]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1

@MikeSpreitzer
Copy link
Member Author

Next I guessed that maybe the memory shortage is on the host rather than inside the VM. So I rebooted the VM, making its memory half as big. Still the problem persists.

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ make clean
+++ [0505 05:46:20] Verifying Prerequisites....

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ ls -ld _output
ls: cannot access '_output': No such file or directory

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 05:47:51] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 05:49:41] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen (non-static)
+++ [0505 05:49:52] Generating prerelease lifecycle code for 26 targets
+++ [0505 05:50:05] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/deepcopy-gen (non-static)
+++ [0505 05:50:11] Generating deepcopy code for 236 targets
+++ [0505 05:50:42] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen (non-static)
+++ [0505 05:50:46] Generating defaulter code for 92 targets
+++ [0505 05:51:17] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/conversion-gen (non-static)
+++ [0505 05:51:21] Generating conversion code for 129 targets
+++ [0505 05:52:16] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/kube-openapi/cmd/openapi-gen (non-static)
+++ [0505 05:52:31] Generating openapi code for KUBE
+++ [0505 05:53:09] Generating openapi code for AGGREGATOR
+++ [0505 05:53:13] Generating openapi code for APIEXTENSIONS
+++ [0505 05:53:18] Generating openapi code for CODEGEN
+++ [0505 05:53:22] Generating openapi code for SAMPLEAPISERVER
+++ [0505 05:53:35] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ /bin/ls -l _output/bin/*
-rwxr-xr-x 1 mspreitz mspreitz 102326272 May  5 06:10 _output/bin/cloud-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz   6361088 May  5 05:51 _output/bin/conversion-gen
-rwxr-xr-x 1 mspreitz mspreitz   6090752 May  5 05:50 _output/bin/deepcopy-gen
-rwxr-xr-x 1 mspreitz mspreitz   6094848 May  5 05:50 _output/bin/defaulter-gen
-rwxr-xr-x 1 mspreitz mspreitz   2314240 May  5 05:48 _output/bin/go2make
-rwxr-xr-x 1 mspreitz mspreitz 125861888 May  5 06:10 _output/bin/kube-apiserver
-rwxr-xr-x 1 mspreitz mspreitz 115507200 May  5 06:10 _output/bin/kube-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz  45707264 May  5 06:10 _output/bin/kubectl
-rwxr-xr-x 1 mspreitz mspreitz 116345384 May  5 06:10 _output/bin/kubelet
-rwxr-xr-x 1 mspreitz mspreitz  41758720 May  5 06:10 _output/bin/kube-proxy
-rwxr-xr-x 1 mspreitz mspreitz  47144960 May  5 06:10 _output/bin/kube-scheduler
-rwxr-xr-x 1 mspreitz mspreitz   8388608 May  5 05:52 _output/bin/openapi-gen
-rwxr-xr-x 1 mspreitz mspreitz   6066176 May  5 05:49 _output/bin/prerelease-lifecycle-gen

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 06:14:11] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 06:15:10] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
# k8s.io/kubernetes/cmd/kube-controller-manager
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/3e/3e15b1d0424f0fffc065ec59b208b1cdd3724e0aa431763672968f17123498a8-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/3e/3e15b1d0424f0fffc065ec59b208b1cdd3724e0aa431763672968f17123498a8-d: cannot allocate memory
# k8s.io/kubernetes/cmd/kube-apiserver
/usr/local/go/pkg/tool/linux_amd64/link: cannot open file /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/bf/bf0b8ee81cf386fbd94770394ebf6ea2acdd019f98721b6ca553c8cb47e192e4-d: open /home/mspreitz/go/src/k8s.io/kubernetes/_output/local/go/cache/bf/bf0b8ee81cf386fbd94770394ebf6ea2acdd019f98721b6ca553c8cb47e192e4-d: cannot allocate memory
!!! [0505 06:15:45] Call tree:
!!! [0505 06:15:45]  1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:716 kube::golang::build_some_binaries(...)
!!! [0505 06:15:45]  2: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/golang.sh:885 kube::golang::build_binaries_for_platform(...)
!!! [0505 06:15:45]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 06:15:45] Call tree:
!!! [0505 06:15:45]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0505 06:15:45] Call tree:
!!! [0505 06:15:45]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:94: all] Error 1

mspreitz@ubu22:~/go/src/k8s.io/kubernetes$ head -5 /proc/meminfo 
MemTotal:       12859640 kB
MemFree:        10463872 kB
MemAvailable:   12215912 kB
Buffers:           49348 kB
Cached:          1892352 kB

@MikeSpreitzer
Copy link
Member Author

I also tried staying inside the guest, not relying on the mount from the host's filesystem. Voila! Problem gone. With guest at 12 GiB of memory, 4 vCPUs, GOPATH set to a parent directory on the guest's main filesystem. I can repeated make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler" without a problem.

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ git checkout release-1.24
Branch 'release-1.24' set up to track remote branch 'release-1.24' from 'origin'.
Switched to a new branch 'release-1.24'

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ git rev-parse HEAD
4e82620fad0b6d57b2e980dcc7f4f537d812b82d

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ ls -ld _output
ls: cannot access '_output': No such file or directory

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ printenv | grep GO
GOPATH=/home/mspreitz/go2

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ pwd
/home/mspreitz/go2/src/k8s.io/kubernetes

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 06:29:27] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 06:29:52] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen (non-static)
+++ [0505 06:30:03] Generating prerelease lifecycle code for 26 targets
+++ [0505 06:30:07] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/deepcopy-gen (non-static)
+++ [0505 06:30:10] Generating deepcopy code for 236 targets
+++ [0505 06:30:21] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/defaulter-gen (non-static)
+++ [0505 06:30:22] Generating defaulter code for 92 targets
+++ [0505 06:30:36] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/code-generator/cmd/conversion-gen (non-static)
+++ [0505 06:30:39] Generating conversion code for 129 targets
+++ [0505 06:31:10] Building go targets for linux/amd64
    k8s.io/kubernetes/vendor/k8s.io/kube-openapi/cmd/openapi-gen (non-static)
+++ [0505 06:31:23] Generating openapi code for KUBE
+++ [0505 06:31:53] Generating openapi code for AGGREGATOR
+++ [0505 06:31:55] Generating openapi code for APIEXTENSIONS
+++ [0505 06:31:58] Generating openapi code for CODEGEN
+++ [0505 06:32:01] Generating openapi code for SAMPLEAPISERVER
+++ [0505 06:32:04] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ date; ls -ld _output/bin/*
Thu May  5 06:50:19 AM UTC 2022
-rwxr-xr-x 1 mspreitz mspreitz 102326272 May  5 06:47 _output/bin/cloud-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz   6361088 May  5 06:30 _output/bin/conversion-gen
-rwxr-xr-x 1 mspreitz mspreitz   6090752 May  5 06:30 _output/bin/deepcopy-gen
-rwxr-xr-x 1 mspreitz mspreitz   6094848 May  5 06:30 _output/bin/defaulter-gen
-rwxr-xr-x 1 mspreitz mspreitz   2314240 May  5 06:29 _output/bin/go2make
-rwxr-xr-x 1 mspreitz mspreitz 125861888 May  5 06:47 _output/bin/kube-apiserver
-rwxr-xr-x 1 mspreitz mspreitz 115507200 May  5 06:47 _output/bin/kube-controller-manager
-rwxr-xr-x 1 mspreitz mspreitz  45707264 May  5 06:47 _output/bin/kubectl
-rwxr-xr-x 1 mspreitz mspreitz 116341256 May  5 06:47 _output/bin/kubelet
-rwxr-xr-x 1 mspreitz mspreitz  41758720 May  5 06:47 _output/bin/kube-proxy
-rwxr-xr-x 1 mspreitz mspreitz  47144960 May  5 06:47 _output/bin/kube-scheduler
-rwxr-xr-x 1 mspreitz mspreitz   8388608 May  5 06:31 _output/bin/openapi-gen
-rwxr-xr-x 1 mspreitz mspreitz   6066176 May  5 06:30 _output/bin/prerelease-lifecycle-gen

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ make WHAT="cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-scheduler"
+++ [0505 06:50:26] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [0505 06:50:32] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ 
mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              1.3G  1.2M  1.3G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   96G   17G   75G  18% /
tmpfs                              6.2G     0  6.2G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
go                                 932G  619G  314G  67% /home/mspreitz/go
/dev/sda2                          2.0G  126M  1.7G   7% /boot
tmpfs                              1.3G  4.0K  1.3G   1% /run/user/1000
tmpfs                              1.3G  4.0K  1.3G   1% /run/user/0

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ head -5 /proc/meminfo 
MemTotal:       12859640 kB
MemFree:         4669696 kB
MemAvailable:   12189604 kB
Buffers:           90668 kB
Cached:          7412792 kB

mspreitz@ubu22:~/go2/src/k8s.io/kubernetes$ go env 
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mspreitz/.cache/go-build"
GOENV="/home/mspreitz/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mspreitz/go2/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mspreitz/go2"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mspreitz/go2/src/k8s.io/kubernetes/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2556406733=/tmp/go-build -gno-record-gcc-switches"

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 3, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 2, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 2, 2022
@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/usability Categorizes an issue or PR as relevant to SIG Usability.
Projects
None yet
Development

No branches or pull requests

3 participants