Cilium no longer runs on Bottlerocket Linux as of 1.11.3

[alex-berger]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

With the changes introduced in #18897 the Cilium agent no longer starts on Bottlerocket Linux nodes.

This is because the node-init DaemonSet cannot write the timestamp to the bootstrapFile ("/tmp/cilium-bootstrap-time") in https://github.com/cilium/cilium/blob/v1.11.3/install/kubernetes/cilium/files/nodeinit/startup.bash#L123 as there is no date command available on Bottlerocket Linux. Instead, it writes an empty bootstrapFile, which in turn causes the test for a non-empty bootstrapFile at https://github.com/cilium/cilium/blob/v1.11.3/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml#L386 to fail.

Due to the above described problem, the agent is stuck forever waiting for the bootstrapFile to become non-empty, which will never happen.

Cilium Version

1.11.3

Kernel Version

5.10.102

Kubernetes Version

1.21

Sysdump

No response

Relevant log output

Waiting on node-init to run...
Waiting on node-init to run...
Waiting on node-init to run...
Waiting on node-init to run...
Waiting on node-init to run...
...

Anything else?

Looks like this is related to #15393 and that the same work-around applies, disabling the node-init DaemonSet.

Code of Conduct

• I agree to follow this project's Code of Conduct

[pchaigno]

cc @aanm

[aanm]

@alex-berger what's the output of ls -la /bin from a Bottlerocket Linux node?

[alex-berger]

@aanm bottlerocket has no tools on the host itself, not even ls nor is there a shell. Everything is inside a container on bottlerocket.

[ubaniabalogun]

Hey is this now a known issue? I'm experiencing errors install Cilium in a bottlerocket EKS node group as well and would like to know if I should debug harder or stop debugging because its broken. Thanks,

[DamiaPoquet]

I'm experiencing the same issue. @aanm is it a confirmed bug already?

[aanm]

@ubaniabalogun @DamiaPoquet correct this is a confirmed bug

[carlosjgp]

You don't need node-init (mount bpf FS) when using BottleRocket just disable it

nodeinit:
  enabled: false # false when using bottlerocket

Edit:

for completeness
bottlerocket-os/bottlerocket#1405 (comment)

[christarazi]

We expect to ship a fix for this in v1.13.2.

[nebril]

This has been fixed as of 1.13.2, please refer to this comment for details: #15393 (comment)