Fails to build 1.21.0 node image - docker buildx now required
#2188
Assignees
Labels
kind/documentation
Categorizes issue or PR as related to documentation.
Comments
yes, the plugin is required.
i think for the time being the requirement should be documented in the kind node image docs. |
|
/remove-kind bug |
k8s-ci-robot
added
kind/documentation
|
Okay, so once I installed the buildx plugin I was able to build the node image, however it fails to start: Click here$ kind create cluster --name v1.21 --image 'kindest/node:v1.21'
Loaded image: kindest/node:v1.21.0
Creating cluster "v1.21" ...
✓ Ensuring node image (kindest/node:v1.21.0) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged v1.21-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0408 21:33:41.777965 248 initconfiguration.go:246] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
I0408 21:33:41.789251 248 kubelet.go:259] setting the KubeletConfiguration cgroupDriver to "systemd"
[init] Using Kubernetes version: v1.21.0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0408 21:33:41.789357 248 certs.go:110] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0408 21:33:41.957924 248 certs.go:487] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost v1.21-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
I0408 21:33:42.469189 248 certs.go:110] creating a new certificate authority for front-proxy-ca
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
I0408 21:33:42.602751 248 certs.go:487] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0408 21:33:42.757067 248 certs.go:110] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0408 21:33:42.908739 248 certs.go:487] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost v1.21-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost v1.21-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0408 21:33:43.599110 248 certs.go:76] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0408 21:33:43.927559 248 kubeconfig.go:101] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0408 21:33:44.299408 248 kubeconfig.go:101] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0408 21:33:44.716312 248 kubeconfig.go:101] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0408 21:33:44.875861 248 kubeconfig.go:101] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0408 21:33:45.208758 248 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0408 21:33:45.291372 248 manifests.go:96] [control-plane] getting StaticPodSpecs
I0408 21:33:45.291840 248 certs.go:487] validating certificate period for CA certificate
I0408 21:33:45.291943 248 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0408 21:33:45.291954 248 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.291960 248 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0408 21:33:45.291966 248 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.291974 248 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.302742 248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I0408 21:33:45.302766 248 manifests.go:96] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0408 21:33:45.303107 248 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0408 21:33:45.303119 248 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.303125 248 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0408 21:33:45.303132 248 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0408 21:33:45.303138 248 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0408 21:33:45.303144 248 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.303151 248 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.304293 248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I0408 21:33:45.304310 248 manifests.go:96] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0408 21:33:45.304601 248 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0408 21:33:45.305220 248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0408 21:33:45.306161 248 local.go:74] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0408 21:33:45.306176 248 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
I0408 21:33:45.307220 248 loader.go:372] Config loaded from file: /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0408 21:33:45.308956 248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s in 0 milliseconds
I0408 21:33:45.810371 248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s in 0 milliseconds
[...]
I0408 21:34:24.809786 248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s in 0 milliseconds
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0408 21:34:25.310322 248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s in 0 milliseconds
[...]
I0408 21:35:40.310530 248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:152
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:152
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
|
|
related to #1969
|
|
I don't have this issue deploying a v1.20 cluster with kind 0.10.0, for which the |
|
I guess v1.21 changed to require cgroups v2? That's the only thing that makes sense. If that's the case, then yes my distro doesn't have the file it's looking for. The kubelet service unit has an which means the script is entirely skipped on cgroup v1 hosts. If the script was run, it would've produced a more descriptive error: if [[ ! -f "/sys/fs/cgroup/cgroup.controllers" ]]; then
echo 'ERROR: this script should not be called on cgroup v1 hosts' >&2
exit 1
fiIf v1.21 requires cgroups v2, it would be useful to run that script unguarded so that the error can be seen in the journal. |
|
could you please log a separate issue and provide the details there? |
Arnavion
changed the title
docker buildx now required
|
Done. #2189 |
|
1.21 requires KIND @ HEAD due to the breaking change in upstream Kubernetes. Similarly the build requirement for buildx comes entirely from Kubernetes upstream. Cgroups v2 is not required, and that script is only for v2. |
|
The breaking cgroups change is kubeadm defaulting kubelet to systemd cgroup driver instead of cgroupsfs in 1.21 which is an action required change for cluster administrators. Kind handles this, but not in v0.10 as when v0.10 was released this was not necessary / known. |
|
In general if you wish to run Kubernetes newer than the kind binary you are using it may not work due to changes like this. We will be releasing in the next week or so after #2176 |
Merged
|
Was there an unexpected issue with 2176? If a release with 1.21 support going to be a while longer, is HEAD likely to work w/1.21 on non-ARM? |
|
I've had some personal / work reasons that I've sort of just been responding to issues / discussions and keeping after reviews, not working on bug fixes / features for a little bit. I'll be back to it. Also the next round of upstream Kubernetes patch releases will fix a major regression in 1.18.x ... 1.21.x where the current patch releases startup significantly slower than before, so we'd kinda like to release alongside those. |
|
In case anybody reaches here trying to build the image with diff --git a/build/lib/release.sh b/build/lib/release.sh
index d8fb1f3df2f..8226d826e44 100644
--- a/build/lib/release.sh
+++ b/build/lib/release.sh
@@ -379,10 +379,10 @@ function kube::release::create_docker_images_for_server() {
ln "${binary_file_path}" "${docker_build_path}/${binary_name}"
local build_log="${docker_build_path}/build.log"
- if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" buildx build \
+ if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" build \
-f "${docker_file_path}" \
--platform linux/"${arch}" \
- --load ${docker_build_opts:+"${docker_build_opts}"} \
+ ${docker_build_opts:+"${docker_build_opts}"} \
-t "${docker_image_tag}" \
--build-arg BASEIMAGE="${base_image}" \
--build-arg SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE}" \Gonna investigate if that upstream code can be made somewhat more flexible to support |
|
Buildx is used to ensure extended permissions are persisted and is necessary for the migration towards a rootless control plane. It is also helpful for ensuring cross complication produces correct image metadata. The dockerfiles also leverage env it populates to streamline them. I am a maintainer of the upstream build and can confidently say we're not interested in losing that, the Kubernetes build process requires a recent install of docker. It also requires a recent bash, and gnu coreutils instead of bsd / Darwin. It doesn't work on windows either. It's just not practical to do what we need otherwise. |
|
See for example past discussion here including another maintainer at RedHat: kubernetes/kubernetes#79048 (comment) This is also a little off-topic for this repo though and discussing upstream build changes upstream will bring visibility to more of the maintainers. This issue here is only open until we make a clear note in our own docs that Kubernetes's source build dependencies are defined by upstream and pointing to the upstream docs for this. |
|
I can confirm that HEAD kind appears to work flawlessly with k8s 1.21 |
|
@BenTheElder understood, thanks for the response! I just left this here in case anybody else was stumbling and needed a potential workaround. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
The
kindcommand fails with:Click here
Something is calling
docker -f ..., preumably it meant to calldocker <subcommand> -fWhat you expected to happen:
Expected it to succeed.
How to reproduce it (as minimally and precisely as possible):
As above.
Anything else we need to know?:
This works with other Kubernetes versions from 1.14 through 1.20, so this is something 1.21-specific.
Environment:
kind version): kind v0.10.0 go1.15.7 linux/amd64kubectl version): 1.21.0docker info): Server Version: 20.10.5-ce/etc/os-release): openSUSE Tumbleweed 20210330Edit: strace indicates the problematic command is:
ie
docker buildx build \ -f '/tmp/kubernetes-v1.21.0/build/server-image/kube-apiserver/Dockerfile' \ --platform linux/amd64 \ --load \ --pull \ -t 'k8s.gcr.io/kube-apiserver-amd64:v1.21.0' \ --build-arg 'BASEIMAGE=k8s.gcr.io/build-image/go-runner:v2.3.1-go1.16.1-buster.0' \ --build-arg 'SETCAP_IMAGE=k8s.gcr.io/build-image/setcap:buster-v1.4.0' \ --build-arg 'BINARY=kube-apiserver' \ '/tmp/kubernetes-v1.21.0/_output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.dockerbuild'Running that command by itself also has the same problem.
Edit 2: Okay, the problem is my distro-provided docker CLI does not have the
buildxplugin.docker foo build -fproduces this confusing error whenfoocan't be found as a plugin.Is this something that can be fixed by
kind(given it's new with 1.21 and was not needed for 1.20 and earlier) ? If not, it needs to be documented somewhere that this docker CLI plugin is required to build node images.The text was updated successfully, but these errors were encountered: