cilium 1.16 enable envoy by default

[albundy83]

Environmental Info:
RKE2 Version:
gbellongervais@localhost:~> kubectl version
Client Version: v1.28.13
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.13+rke2r1

Cluster Configuration:
3 control-planes
2 workers

Describe the bug:
If I don't specify to disable Envoy deployment, I have Envoy deployed as DaemonSet, which is the new correct behaviour according to cilium doc cilium/cilium@471f19a

I need to specify:

envoy:
  enabled: false

I don't know if it's really a bug or a new feature but maybe we could document this somewhere.

[albundy83]

Here my full config to deploy cilium with network policies enabled:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  bootstrap: true
  valuesContent: |
    cluster:
      name: my-lovely-kube
      id: 1
    kubeProxyReplacement: true
    k8sServiceHost: my-kubernetes-load-balancer-hostname.fr
    k8sServicePort: 6443
    envoy:
      enabled: false   # Now it's mandatory or you have new cilium-envoy DaemonSet
    cni:
      chainingMode: none
    bpf:
      masquerade: true
    MTU: 1500
    encryption:
      enabled: false
      type: wireguard
      nodeEncryption: false
    hubble:
      enabled: true
      metrics:
        enabled:
          - dns
          - drop
          - tcp
          - flow
          - port-distribution
          - icmp
          - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
        serviceMonitor:
          enabled: true
          relabelings:
            - sourceLabels:
                - __meta_kubernetes_pod_node_name
              targetLabel: node
              replacement: ${1}
            - targetLabel: cluster
              replacement: my-lovely-kube
              action: replace
        dashboards:
          enabled: true
      relay:
        enabled: true
        prometheus:
          enabled: true
          serviceMonitor:
            enabled: true
            relabelings:
              - targetLabel: cluster
                replacement: my-lovely-kube
                action: replace
      ui:
        enabled: true
        ingress:
          enabled: true
          annotations:
            cert-manager.io/cluster-issuer: letsencrypt-prod
          className: nginx
          hosts:
            - hubble.my-lovely-hostname.fr
          tls:
            - secretName: hubble.my-lovely-hostname.fr
              hosts:
                - hubble.my-lovely-hostname.fr
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
        relabelings:
          - sourceLabels:
              - __meta_kubernetes_pod_node_name
            targetLabel: node
            replacement: ${1}
          - targetLabel: cluster
            replacement: my-lovely-kube
            action: replace
    dashboards:
      enabled: true
    operator:
      enabled: true
      rollOutPods: true
      prometheus:
        enabled: true
        serviceMonitor:
          enabled: true
          relabelings:
            - targetLabel: cluster
              replacement: my-lovely-kube
              action: replace
      dashboards:
        enabled: true
    
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: prometheus-to-hubble
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: hubble
  ingress:
    - ports:
        - protocol: TCP
          port: hubble-metrics
      from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/instance: kube-prometheus-stack-prometheus
              app.kubernetes.io/name: prometheus
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: prometheus-to-hubble-relay
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: hubble-relay
  ingress:
    - ports:
        - protocol: TCP
          port: prometheus
      from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/instance: kube-prometheus-stack-prometheus
              app.kubernetes.io/name: prometheus
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
  policyTypes:
    - Ingress

[VestigeJ]

##Environment Details
Reproduced using VERSION=v1.31.1+rke2r1
Validated using COMMIT=

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-1-1-1-7         Ready    control-plane,etcd,master   16m   v1.31.1+rke2r1

Config.yaml:

node-external-ip: 1.1.1.7
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cni: cilium
embedded-registry: true

Details

Details

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.31.1+rke2r1
$ sudo INSTALL_RKE2_VERSION=$VERSION INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ go_rke2
$ kd po cilium-jsccd -n kube-system | grep -i image

Results:

Current release image versions used by cilium pods already contain this version even though the milestone/release was set for October.

$ kd po cilium-jsccd -n kube-system | grep -i image

    Image:          rancher/hardened-cni-plugins:v1.5.1-build20240910
    Image ID:       docker.io/rancher/hardened-cni-plugins@sha256:1e37bdbc042ac18ca8c4e336b8a33bde31b1798ad9eb5274b7de1ff9d34a52f7
    Image:         rancher/mirrored-cilium-cilium:v1.16.1
    Image ID:      docker.io/rancher/mirrored-cilium-cilium@sha256:0b4a3ab41a4760d86b7fc945b8783747ba27f29dac30dd434d94f2c9e3679f39
    Image:         rancher/mirrored-cilium-cilium:v1.16.1

Confirming that the image is present via commit_id on all active branches

v1.31 COMMIT_ID=5f70d18fe7637135995a621627983f791c1fc3da

kd po cilium-8kff2 -n kube-system | grep -i image
    Image:          rancher/hardened-cni-plugins:v1.5.1-build20240910
    Image ID:       docker.io/rancher/hardened-cni-plugins@sha256:1e37bdbc042ac18ca8c4e336b8a33bde31b1798ad9eb5274b7de1ff9d34a52f7
    Image:         rancher/mirrored-cilium-cilium:v1.16.1
    Image ID:      docker.io/rancher/mirrored-cilium-cilium@sha256:0b4a3ab41a4760d86b7fc945b8783747ba27f29dac30dd434d94f2c9e3679f39

v1.30 COMMIT_ID=609736bbe10fcb2b32d613ffb2329b089174dbf3

 kd po cilium-ggplj -n kube-system | grep -i image
    Image:          rancher/hardened-cni-plugins:v1.5.1-build20240910
    Image ID:       docker.io/rancher/hardened-cni-plugins@sha256:1e37bdbc042ac18ca8c4e336b8a33bde31b1798ad9eb5274b7de1ff9d34a52f7
    Image:         rancher/mirrored-cilium-cilium:v1.16.1
    Image ID:      docker.io/rancher/mirrored-cilium-cilium@sha256:0b4a3ab41a4760d86b7fc945b8783747ba27f29dac30dd434d94f2c9e3679f39

v1.29 COMMIT_ID=78e173a47c228286320e7967f2df62635aba795d

kd po cilium-n6tbt -n kube-system | grep -i image
    Image:          rancher/hardened-cni-plugins:v1.5.1-build20240910
    Image ID:       docker.io/rancher/hardened-cni-plugins@sha256:1e37bdbc042ac18ca8c4e336b8a33bde31b1798ad9eb5274b7de1ff9d34a52f7
    Image:         rancher/mirrored-cilium-cilium:v1.16.1
    Image ID:      docker.io/rancher/mirrored-cilium-cilium@sha256:0b4a3ab41a4760d86b7fc945b8783747ba27f29dac30dd434d94f2c9e3679f39

v1.28 COMMIT_ID=bab888a04e223189d39c7342c191f7ce592c3443

kd po cilium-vlcqj -n kube-system | grep -i image
    Image:          rancher/hardened-cni-plugins:v1.5.1-build20240910
    Image ID:       docker.io/rancher/hardened-cni-plugins@sha256:1e37bdbc042ac18ca8c4e336b8a33bde31b1798ad9eb5274b7de1ff9d34a52f7
    Image:         rancher/mirrored-cilium-cilium:v1.16.1
    Image ID:      docker.io/rancher/mirrored-cilium-cilium@sha256:0b4a3ab41a4760d86b7fc945b8783747ba27f29dac30dd434d94f2c9e3679f39