Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minikube addons enable ingress fails on Minikube v1.33.0 #18738

Closed
mhrheaume opened this issue Apr 24, 2024 · 7 comments · Fixed by #18779
Closed

minikube addons enable ingress fails on Minikube v1.33.0 #18738

mhrheaume opened this issue Apr 24, 2024 · 7 comments · Fixed by #18779

Comments

@mhrheaume
Copy link

What Happened?

This is on a brand new Minikube:

$ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]

Looking at the pod it appears there is a problem pulling the image:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-84df5799c-ld4hn
Name:             ingress-nginx-controller-84df5799c-ld4hn
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.6
Start Time:       Tue, 23 Apr 2024 20:13:36 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=84df5799c
Annotations:      <none>
Status:           Pending
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-84df5799c
Containers:
  controller:
    Container ID:
    Image:         registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Image ID:
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-84df5799c-ld4hn (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lqq2s (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-lqq2s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    12m                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-84df5799c-ld4hn to minikube
  Warning  FailedMount  12m (x3 over 12m)    kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      10m (x4 over 12m)    kubelet            Pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"
  Warning  Failed       10m (x4 over 11m)    kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c": failed to register layer: lsetxattr security.capability /nginx-ingress-controller: operation not supported
  Warning  Failed       10m (x4 over 11m)    kubelet            Error: ErrImagePull
  Warning  Failed       10m (x5 over 11m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff      119s (x39 over 11m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

QEMU
@mhrheaume
Copy link
Author

FYI, I just tried this on v1.32.0 and it works:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-7c6974c4d8-wcm5v
Name:             ingress-nginx-controller-7c6974c4d8-wcm5v
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.7
Start Time:       Tue, 23 Apr 2024 20:32:21 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=7c6974c4d8
Annotations:      <none>
Status:           Running
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  docker://a77144fb810e8b5f6abb892471c476f3ba61942b8efcbd352ce04a2fa99798c7
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 23 Apr 2024 20:32:31 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7c6974c4d8-wcm5v (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fx9dm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-fx9dm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From                      Message
  ----     ------       ----               ----                      -------
  Normal   Scheduled    44s                default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-wcm5v to minikube
  Warning  FailedMount  43s (x3 over 44s)  kubelet                   MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      40s                kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3"
  Normal   Pulled       34s                kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" in 6.373s (6.373s including waiting)
  Normal   Created      34s                kubelet                   Created container controller
  Normal   Started      34s                kubelet                   Started container controller
  Normal   RELOAD       32s                nginx-ingress-controller  NGINX reload triggered due to a change in configuration

@humcqc
Copy link

humcqc commented Apr 24, 2024

seems telepresence helm install have the same issue

@medyagh
Copy link
Member

medyagh commented Apr 30, 2024 •
edited

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.

 minikube start --driver=qemu2 --container-runtime=containerd

@spowelljr spowelljr mentioned this issue Apr 30, 2024
Merged
@medyagh medyagh mentioned this issue Apr 30, 2024
Closed
@spowelljr
Copy link
Member

spowelljr commented Apr 30, 2024 •
edited

Based on AdguardTeam/AdGuardHome#6816, it seems like the issue should be specific to users using a VM driver on arm64. We don't have reliable VM arm64 tests so this would explain how this went undetected.

We have CONFIG_EXT4_FS_SECURITY enabled on amd64:

minikube/deploy/iso/minikube-iso/board/minikube/x86_64/linux_x86_64_defconfig

Line 483 in 86fc9d5

CONFIG_EXT4_FS_SECURITY=y

But it's missing from arm64, created #18779 to add it

@mhrheaume
Copy link
Author

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.

 minikube start --driver=qemu2 --container-runtime=containerd

My driver is also QEMU arm64 (M2 MacBook Pro).

@spowelljr
Copy link
Member

spowelljr commented May 1, 2024 •
edited

Hi @mhrheaume, I've confirmed that my fix resolves the issue. We're going to make a patch release in the future but we don't have an exact timeline for it currently. In the meantime, you can pass this iso-url flag on start that will use the ISO from my fix and resolve your issue until the release is out.

minikube start --driver qmeu --iso-url="https://storage.googleapis.com/minikube-builds/iso/18779/minikube-v1.33.0-1714498396-18779-arm64.iso"

@spowelljr
Copy link
Member

Fix is included in latest release of minikube: https://github.com/kubernetes/minikube/releases/tag/v1.33.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ISO: enable CONFIG_EXT4_FS_SECURITY in arm64 spowelljr/minikube
4 participants
@mhrheaume @medyagh @spowelljr @humcqc