Cilium pod got `Init:CreateContainerError` on Ubuntu 24.04, k8s 1.30 #32198

orimanabu · 2024-04-26T07:45:07Z

Is there an existing issue for this?

I have searched the existing issues

What happened?

Cilium pod does not start and goes into Init:CreateContainerError.
kubectl describe pod says:

Error: applying apparmor profile to container 43ed6b4ba299559e8eac46a32f3246d9c54aca71a9b460576828b662147558fa: empty localhost AppArmor profile is forbidden

Environment:

k8s v1.30
Ubuntu 24.04
cilium v1.15.3
CRI-O v1.30-dev

Note: control plane is Fedora 40.

$ kubectl get node -o wide
NAME                          STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION          CONTAINER-RUNTIME
k8s-129-cp1.fedora-amd.home   Ready    control-plane   51m   v1.30.0   172.16.190.11   <none>        Fedora Linux 40 (Server Edition)   6.8.7-300.fc40.x86_64   cri-o://1.30.0
k8s-129-wk1.fedora-amd.home   Ready    <none>          50m   v1.30.0   172.16.190.21   <none>        Fedora Linux 40 (Server Edition)   6.8.7-300.fc40.x86_64   cri-o://1.30.0
k8s-129-wk2.fedora-amd.home   Ready    <none>          50m   v1.30.0   172.16.190.22   <none>        Fedora Linux 40 (Server Edition)   6.8.7-300.fc40.x86_64   cri-o://1.30.0
k8s-c9s.fedora-amd.home       Ready    <none>          27m   v1.30.0   172.16.190.31   <none>        CentOS Stream 9                    5.14.0-437.el9.x86_64   cri-o://1.30.0
k8s-ubuntu.fedora-amd.home    Ready    <none>          16m   v1.30.0   172.16.190.41   <none>        Ubuntu 24.04 LTS                   6.8.0-31-generic        cri-o://1.30.0

$ kubectl get pod -o wide -A
NAMESPACE     NAME                                                  READY   STATUS                      RESTARTS   AGE   IP              NODE                          NOMINATED NODE   READINESS GATES
kube-system   cilium-7q7kq                                          1/1     Running                     0          27m   172.16.190.22   k8s-129-wk2.fedora-amd.home   <none>           <none>
kube-system   cilium-8d7tf                                          0/1     Init:CreateContainerError   0          17m   172.16.190.41   k8s-ubuntu.fedora-amd.home    <none>           <none>
kube-system   cilium-97fwm                                          1/1     Running                     0          27m   172.16.190.21   k8s-129-wk1.fedora-amd.home   <none>           <none>
kube-system   cilium-kfn4q                                          1/1     Running                     0          27m   172.16.190.31   k8s-c9s.fedora-amd.home       <none>           <none>
kube-system   cilium-operator-56477b846b-hxzbn                      1/1     Running                     0          27m   172.16.190.22   k8s-129-wk2.fedora-amd.home   <none>           <none>
kube-system   cilium-wg2xd                                          1/1     Running                     0          27m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
kube-system   coredns-7db6d8ff4d-8wsbv                              1/1     Running                     0          27m   10.0.0.100      k8s-129-wk1.fedora-amd.home   <none>           <none>
kube-system   coredns-7db6d8ff4d-wwv9g                              1/1     Running                     0          27m   10.0.0.175      k8s-129-wk1.fedora-amd.home   <none>           <none>
kube-system   etcd-k8s-129-cp1.fedora-amd.home                      1/1     Running                     0          51m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
kube-system   kube-apiserver-k8s-129-cp1.fedora-amd.home            1/1     Running                     0          51m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
kube-system   kube-controller-manager-k8s-129-cp1.fedora-amd.home   1/1     Running                     0          51m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
kube-system   kube-proxy-485rk                                      1/1     Running                     0          51m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
kube-system   kube-proxy-4vk9r                                      1/1     Running                     0          17m   172.16.190.41   k8s-ubuntu.fedora-amd.home    <none>           <none>
kube-system   kube-proxy-5dfft                                      1/1     Running                     0          50m   172.16.190.22   k8s-129-wk2.fedora-amd.home   <none>           <none>
kube-system   kube-proxy-7t2lp                                      1/1     Running                     0          50m   172.16.190.21   k8s-129-wk1.fedora-amd.home   <none>           <none>
kube-system   kube-proxy-n4bqf                                      1/1     Running                     0          27m   172.16.190.31   k8s-c9s.fedora-amd.home       <none>           <none>
kube-system   kube-scheduler-k8s-129-cp1.fedora-amd.home            1/1     Running                     0          51m   172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>

Cilium Version

cilium-cli: v0.16.4 compiled with go1.22.1 on linux/amd64
cilium image (default): v1.15.3
cilium image (stable): v1.15.4
cilium image (running): 1.15.3

Kernel Version

6.8.0-31-generic

Kubernetes Version

Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Regression

No response

Sysdump

cilium-sysdump-20240425-004439.zip

Relevant log output

Events:
  Type     Reason     Age                             From               Message
  ----     ------     ----                            ----               -------
  Normal   Scheduled  14m                             default-scheduler  Successfully assigned kube-system/cilium-8d7tf to k8s-ubuntu.fedora-amd.home
  Normal   Pulling    <invalid>                       kubelet            Pulling image "quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0"
  Normal   Pulled     <invalid>                       kubelet            Successfully pulled image "quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0" in 43.475s (51.122s including waiting). Image size: 587097123 bytes.
  Normal   Created    <invalid>                       kubelet            Created container config
  Normal   Started    <invalid>                       kubelet            Started container config
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 43ed6b4ba299559e8eac46a32f3246d9c54aca71a9b460576828b662147558fa: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 23cfb278f9405a7638b25aa72cd332bb3ba0bfcace981f56cad8d311957620e8: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container e7a8f9a86ed59bb9406d0931947af5ce94bf0dfde3c3d1b38b21c718f15c8e30: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 463156777fa07e31d382fe57e114198ead7b12c869da9f6aa42d425fc6c8c488: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 7236abc883576f1263d4741b26b41dd179eb8b6daf842122559661ed395d0040: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container c26d034f2d9f2be8224e30e6f466dc67b6b02d406fe369fe7ca1b516c872e0e8: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container c7b7e778515992205326e0d323a2666283c47e03ece3ce4dd4222309d1837d98: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container c3cb23460ba2acd972d378b991e84574ad0bb8767252b96f32a7206c917fc615: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 63e3ab881d8849968f88d8361eecccbe0d828fe3d92c674372f48a40d9227e09: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            (combined from similar events): Error: applying apparmor profile to container 541ebb150e4727541d3c68754fc328c63330f7ee0fd91661be06d38eab8d6de9: empty localhost AppArmor profile is forbidden
  Normal   Pulled     <invalid> (x11 over <invalid>)  kubelet            Container image "quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0" already present on machine
  Normal   Pulled     <invalid> (x28 over <invalid>)  kubelet            Container image "quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0" already present on machine

Anything else?

No response

Cilium Users Document

Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

aanm · 2024-04-26T08:24:42Z

@orimanabu thank you for opening the GH issue. Can you try installing Cilium with:

--set podSecurityContext.appArmorProfile.type="Unconfined"

orimanabu · 2024-04-26T08:34:01Z

Thank you for your quick response.
I tried cilium uninstall and cilium install --set podSecurityContext.appArmorProfile.type="Unconfined", but no luck.
The symptom remainds the same...

aanm · 2024-04-26T08:46:01Z

@orimanabu can you send a sysdump for that installation?

orimanabu · 2024-04-26T08:52:09Z

Sure!
cilium-sysdump-20240425-020741.zip

aanm · 2024-04-26T12:53:27Z

Sure! cilium-sysdump-20240425-020741.zip

@orimanabu That sysdump doesn't containt he app armor in the daemonset indeed. I've tried it out in a kind cluster running with k8s 1.30 and I it seems that it's working for me:

$ kind  create  cluster --image quay.io/cilium/kindest-node:v1.30.0
Creating cluster "kind" ...
 ✓ Ensuring node image (quay.io/cilium/kindest-node:v1.30.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

$ cilium install --set podSecurityContext.appArmorProfile.type="Unconfined" 
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.22.0"
ℹ  Using Cilium version 1.14.1
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected kube-proxy has been installed
$ kubectl get ds -n kube-system cilium   -o yaml | grep Armor -A 1
        appArmorProfile:
          type: Unconfined

orimanabu · 2024-04-26T14:01:38Z

I manually edit the DaemonSet and add appArmorProfile, but still no luck.

$ kubectl -n kube-system get ds/cilium -o json | jq .spec.template.spec.securityContext
{
  "appArmorProfile": {
    "type": "Unconfined"
  }
}

$ kubectl -n kube-system get pod -l k8s-app=cilium -o wide
NAME           READY   STATUS                      RESTARTS   AGE     IP              NODE                          NOMINATED NODE   READINESS GATES
cilium-2nj6x   1/1     Running                     0          11m     172.16.190.11   k8s-129-cp1.fedora-amd.home   <none>           <none>
cilium-dsjsj   1/1     Running                     0          10m     172.16.190.21   k8s-129-wk1.fedora-amd.home   <none>           <none>
cilium-tt6lz   1/1     Running                     0          11m     172.16.190.31   k8s-c9s.fedora-amd.home       <none>           <none>
cilium-txltl   1/1     Running                     0          10m     172.16.190.22   k8s-129-wk2.fedora-amd.home   <none>           <none>
cilium-v5qvr   0/1     Init:CreateContainerError   0          9m44s   172.16.190.41   k8s-ubuntu.fedora-amd.home    <none>           <none>

$ kubectl -n kube-system describe pod cilium-v5qvr | sed -e '1,/^Events:/d'
  Type     Reason     Age                             From               Message
  ----     ------     ----                            ----               -------
  Normal   Scheduled  9m49s                           default-scheduler  Successfully assigned kube-system/cilium-v5qvr to k8s-ubuntu.fedora-amd.home
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container b456dce1bd54bab72c8c4a3342db0c9c9a2d552b526e62406c069d4e1c8dc277: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container f8d0d1e257a05170d1784db55c93e71621b0ed91e5b417441d949e54aa9c1f3e: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 00543e0e1084190489fe6e6c241a89b2648301e36d5d6d0697c61821a8bc089f: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container df05ca0e5499fd2f362d62d92b65916220047ded834402daac25d5b68df0f2cc: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container f1efffede72f554720fc3462ccfad6a20fb84755a2a38ec1b90f9a5ff394b721: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 018f9d2899034f42d0025ea1d3a3457ab1a8f1a6abf21addbd0448c809af4699: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container f038667a72c06449dc21e111ddc5a0212ee52d4d191c1e6d016d89ef216d5d29: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container bc7bef26db94e08039c9054c4061422ba262169045b57cbaadb200525b0704c2: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid>                       kubelet            Error: applying apparmor profile to container 28cc3788d2bedd7912796cd7ccd6d6014cb01951768839f98391993c9f07f7ab: empty localhost AppArmor profile is forbidden
  Warning  Failed     <invalid> (x3 over <invalid>)   kubelet            (combined from similar events): Error: applying apparmor profile to container fb6ade44bb5326045ea3e46218b3cd238ae70132a605c8405dc70e2c8159bec5: empty localhost AppArmor profile is forbidden
  Normal   Pulled     <invalid> (x25 over <invalid>)  kubelet            Container image "quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0" already present on machine

I'll destroy the cluster and recreate the whole cluster with --set podSecurityContext.appArmorProfile.type="Unconfined" in the first place.

aanm · 2024-04-26T14:33:47Z

can you also try --set podSecurityContext.appArmorProfile.type="unconfined"? (Noticed the small 'u' in unconfined)

est-it · 2024-04-26T14:54:24Z

I'm having the same problem with k8s v1.30.

Writing "unconfined" instead of "Unconfined" doesn't work:

helm install cilium cilium/cilium --version 1.15.4 --namespace kube-system -f cilium.yaml

Error: INSTALLATION FAILED: 1 error occurred:
        * DaemonSet.apps "cilium" is invalid: spec.template.spec.securityContext.appArmorProfile.type: Unsupported value: "unconfined": supported values: "Localhost", "RuntimeDefault", "Unconfined"

I'm on Debian 12.

orimanabu · 2024-04-26T14:55:39Z

Same here, small 'u' didn't work.

$ cilium install --set podSecurityContext.appArmorProfile.type="unconfined"
ℹ️  Using Cilium version 1.15.3
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has been installed

Error: Unable to install Cilium: 1 error occurred:
	* DaemonSet.apps "cilium" is invalid: spec.template.spec.securityContext.appArmorProfile.type: Unsupported value: "unconfined": supported values: "Localhost", "RuntimeDefault", "Unconfined"

est-it · 2024-04-26T15:00:15Z

I found the line that generates the error message. Don't know if it helps to solve the issue.

https://github.com/cri-o/cri-o/blob/b4b0482fc65e3d0b7530b21beab48b547ded65c1/internal/config/apparmor/apparmor_linux.go#L123

As it's a line from CRI-O, I think, the issue only occurs, when using CRI-O as container runtime.

Looks like "securityProfile" isn't set correctly.

I'm on phone at the moment, but I think "Unconfined" isn't recognized correctly.

aanm · 2024-04-26T19:31:25Z

Can you try with --set podSecurityContext.appArmorProfile.type="RuntimeDefault"

est-it · 2024-04-26T22:33:42Z

This works with some additional configuration:

podSecurityContext:
  appArmorProfile:
    type: "RuntimeDefault"
podAnnotations:
  "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "runtime/default"
  "container.apparmor.security.beta.kubernetes.io/cilium-agent": "runtime/default"
  "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "runtime/default"
  "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "runtime/default"

Additionally I had to set "Unconfined" as CRI-O default AppArmor profile, else the agents fail.

/etc/crio/crio.conf.d/20-apparmordefault.conf:
[crio.runtime]
apparmor_profile="unconfined"

As this affects the whole node, I wouldn't recommend it for production use.

aanm · 2024-04-26T23:15:15Z

So cri-o doesn't have a default app armor profile set?

est-it · 2024-04-26T23:29:05Z

It has:

--apparmor-profile="": Name of the apparmor profile to be used as the runtime's default. This only takes effect if the user does not specify a profile via the Kubernetes Pod's metadata annotation. (default: "crio-default")

https://github.com/cri-o/cri-o/blob/main/docs/crio.8.md

But the profile blocks some things that cilium needs (some /proc things). So the cilium agent crashes.
The above configuration "works", but of course has potential for big impact.

jadedeane · 2024-05-08T03:13:28Z

Cilium 1.16.0-pre.2 and CRI-O 1.30.0~dev-84.1 (Ubuntu 24.04, K8s 1.30), cilium ds w/"unconfined" annotations as expected:

  annotations:
    container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites: unconfined
    container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined
    container.apparmor.security.beta.kubernetes.io/clean-cilium-state: unconfined
    container.apparmor.security.beta.kubernetes.io/config: unconfined
    container.apparmor.security.beta.kubernetes.io/install-cni-binaries: unconfined
    container.apparmor.security.beta.kubernetes.io/mount-bpf-fs: unconfined
    container.apparmor.security.beta.kubernetes.io/mount-cgroup: unconfined

However, despite Cilium's fix and CRI-O's fix that made it into 1.30.0 release, still seeing:

empty localhost AppArmor profile is forbidden

Only fix seems to be the above.

est-it · 2024-05-08T04:32:33Z

Just set up a new cluster yesterday with latest cilium and cri-o. The issue is fixed for me.

I don't have to set SecurityContexts or annotations for cilium. I'm just using the default values now.

jadedeane · 2024-05-08T07:04:00Z

Resolved into CRI-O 1.31.0~dev-2.1 for me.

orimanabu added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels Apr 26, 2024

aanm mentioned this issue Apr 26, 2024

install/kubernetes: add AppArmor profile to Cilium Daemonset #32199

Merged

aanm added sig/k8s Impacts the kubernetes API, or kubernetes -> cilium internals translation layers. sig/agent Cilium agent related. and removed needs/triage This issue requires triaging to establish severity and next steps. labels Apr 26, 2024

aanm self-assigned this Apr 26, 2024

aanm mentioned this issue Apr 26, 2024

Unable to run pods with apparmor type "Unconfined" cri-o/cri-o#8080

Closed

aanm closed this as completed in #32199 Apr 29, 2024

jadedeane mentioned this issue May 8, 2024

Fix AppArmor profile Apply() function to correctly handle an "Unconfined" mode cri-o/cri-o#8103

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cilium pod got `Init:CreateContainerError` on Ubuntu 24.04, k8s 1.30 #32198

Cilium pod got `Init:CreateContainerError` on Ubuntu 24.04, k8s 1.30 #32198

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024 •

edited

orimanabu commented Apr 26, 2024

est-it commented Apr 26, 2024 •

edited

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024 •

edited

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024

jadedeane commented May 8, 2024 •

edited

est-it commented May 8, 2024

jadedeane commented May 8, 2024

Cilium pod got Init:CreateContainerError on Ubuntu 24.04, k8s 1.30 #32198

Cilium pod got Init:CreateContainerError on Ubuntu 24.04, k8s 1.30 #32198

Comments

orimanabu commented Apr 26, 2024

Is there an existing issue for this?

What happened?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

orimanabu commented Apr 26, 2024

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024 • edited

orimanabu commented Apr 26, 2024

est-it commented Apr 26, 2024 • edited

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024 • edited

aanm commented Apr 26, 2024

est-it commented Apr 26, 2024

jadedeane commented May 8, 2024 • edited

est-it commented May 8, 2024

jadedeane commented May 8, 2024

Cilium pod got `Init:CreateContainerError` on Ubuntu 24.04, k8s 1.30 #32198

Cilium pod got `Init:CreateContainerError` on Ubuntu 24.04, k8s 1.30 #32198

est-it commented Apr 26, 2024 •

edited

est-it commented Apr 26, 2024 •

edited

est-it commented Apr 26, 2024 •

edited

jadedeane commented May 8, 2024 •

edited