Skip to content

1.11.0
Compare
Choose a tag to compare
@joestringer joestringer released this 09 Dec 16:12
v1.11.0
This tag was signed with the committer’s verified signature. The key has expired.
joestringer Joe Stringer
GPG key ID: 849D37862B67CB57
Expired
Learn about vigilant mode.
27e0848

The Cilium core team are excited to announce the Cilium 1.11 release. A total of 1395 new commits have been contributed to this release by a growing community of over 350 developers. 🎉

  • OpenTelemetry Support: Ability to export Hubble's L3-L7 observability data in OpenTelemetry tracing and metrics format
  • Simplified Policy to the Kubernetes API server: New policy entity for hassle-free policy modeling of communication from/to the Kubernetes API server.
  • Topology Aware Load-Balancing: Enhanced load-balancing with support for topology-aware hints to route traffic to the closest endpoint, or to keep traffic within a region.
  • BGP Pod CIDR Announcement: Advertise PodCIDR IP routes to your network using BGP.
  • Graceful Service Backend Termination: Support for graceful connection termination in order to drain network traffic when load-balancing to workloads and Pods that are being terminated.
  • Host Firewall Promotion: Host firewall functionality has been promoted to stable and is ready for production use.
  • Improved Load Balancer Scalability: Cilium load balancing now supports more than 64K backend endpoints.
  • Improved Load Balancer Device Support: The accelerated XDP fast-path for load-balancing can now be used with bonded devices and more generally also in multi-device setups.
  • Compatibility of kube-proxy Replacement with Istio: Cilium's kube-proxy replacement mode is now compatible with Istio sidecar deployments.
  • Egress Gateway Improvements: Enhancements to the egress gateway functionality, including support for additional datapath modes.
  • Managed Neighbor Discovery: Extensions to both the Linux kernel as well as Cilium's load-balancer in order to remove its internal ARP library and delegate the next hop discovery for IPv4 and now also IPv6 nodes to the kernel.
  • Simplified Device Detection: Improved user experience for multi-device setups with Cilium through route-based auto-detection of external-facing network devices.
  • Cgroup v2 Enhancements: Enhancements to Cilium's kube-proxy replacement integration for runtimes operating in pure cgroup v2 mode as well as Linux kernel improvements for Kubernetes mixed mode cgroup v1/v2 environments.
  • Cilium Endpoint Slices: Cilium is now more efficient in CRD mode with its control-plane interactions with Kubernetes, enabling 1000+ node scalability in a way that previously required a dedicated etcd instance to manage.
  • MKE Integration: Support for Mirantis Kubernetes Engine.

Known issues

  • ToFQDN network policy statements may become ineffective for a pod after modifying a network policy that selects the pod, for instance denying traffic that should be allowed by a ToFQDN rule. The mitigation is to apply all policies first and then restart the Cilium agent or restart the affected pods. (#18023)

The summary of changes below reflect the diff between the last stable release v1.10.5 and tag v1.11.0.

Summary of Changes

Major Changes:

  • Add Kubernetes Service Topology Aware Hints (Backport PR #18027, Upstream PR #17929, @brb)
  • Add support for k8s 1.23.0 (Backport PR #18027, Upstream PR #18008, @aanm)
  • Cilium Envoy integration is updated to Envoy release 1.18.4 (#17236, @jrajahalme)
  • Cilium Istio integration is updated to Istio release 1.9.6. (#16766, @jrajahalme)
  • New performance benchmarks and tuning guide (#15943, @tgraf)
  • New CiliumEndpointSlice feature for better scalability in CRD-only clusters (#17658, @krishgobinath)
  • Add ICMP and ICMPv6 support for CNP and CCNP policies with a feature flag (#16516, @chez-shanpu)
  • Provide new installation steps to deploy Cilium in managed kubernetes providers (GKE, EKS, AKS) to allow scale up and down node pools. (#16631, @aanm)
  • Support policy matching against kube-apiserver entity (#17823, @christarazi)
  • Support graceful termination for service load-balancing such that active connections don't break when endpoints are deleted. (#17716, @aditighag)

Minor Changes:

  • allow-any-ingress and allow-remotehost-ingress are now used instead of allow-localhost-ingress in policy rule derivedFrom list when appropriate. (#16972, @jrajahalme)
  • Add flag to list all available configurations (#17303, @h3llix)
  • Add Helm option to disable registering CRD from Cilium Operator (#15655, @Fedosin)
  • Add validation of agent flag values for ConfigMap (#16014, @romanspb80)
  • Add WireGuard status to cilium encrypt. (#17684, @h3llix)
  • Add workload name and workload kind to slim api and hubble api (#16514, @sugangli)
  • Adds new Cilium subcommand: cilium encrypt status and cilium encrypt flush (#16770, @h3llix)
  • Auto discover ipv6-mcast-device if not provided (#16692, @sarveshr7)
  • Auto-detect Azure cloud name via IMDS (#16515, @ungureanuvladvictor)
  • Auto-mount bpf file-system from within Cilium DaemonSet and remove the requirement of having it mounted in the host. (#16656, @aanm)
  • AWS eni: Support Instance Metadata Service Version 2 (IMDSv2) (#15828, @Smana)
  • bpf: Derive host netns cookie via SO_NETNS_COOKIE (#17018, @brb)
  • Cilium Istio integration is updated to Istio release 1.10.3. (#17037, @jrajahalme)
  • cilium: Improve user experience of policy trace with regard to port a… (#15929, @Maddy007-maha)
  • cilium: Make CLI more graceful on environments with IPv6 disabled (#16168, @Maddy007-maha)
  • cleanup helm chart (#16896, @dungdm93)
  • crd: Add categories for cilium CRDs (#17162, @sayboras)
  • daemon: Add option --bpf-lb-external-clusterip (#15650, @joamaki)
  • daemon: Add wildcard support to --devices ("eth+") (#15697, @joamaki)
  • daemon: make consecutive quorum errors threshold configurable (#16885, @ArthurChiao)
  • daemon: Make L2 neighbor discovery configurable. (#16974, @bjhaid)
  • datapath,daemon: Enable multi-dev XDP (#17655, @brb)
  • datapath: Add a flag to set VXLAN and Geneve ports (#16874, @errordeveloper)
  • datapath: Add a new option to skip socket lb when in pod ns (#17154, @brb)
  • datapath: optionally disable SIP verification (#16134, @oblazek)
  • Detect devices from global unicast routes in addition to only looking for the device with the Kubernetes Node IP and the one with default route. This expands the set of devices used for kube-proxy replacement, host firewall and bandwidth manager and should reduce the need to specify devices manually. (#17219, @joamaki)
  • Display host firewall status in cilium status (#17165, @pchaigno)
  • doc: Add more generic install section for egress gateway guide (#16087, @tgraf)
  • doc: Reword some results (#15955, @tgraf)
  • doc: Update diagrams in benchmark report (#16063, @tgraf)
  • docs: Remove firewall hack for OKD GSG (#17924, @errordeveloper)
  • docs: Revert host firewall to beta for kube-proxy setups (#16149, @pchaigno)
  • Envoy is updated to release 1.18.3 (#17024, @jrajahalme)
  • Extend cilium config to expose all active configurations. Add subcommand cilium config get to get configurations from CLI (#16519, @h3llix)
  • feat: allow installing hubble ui as standalone (#17473, @eddycharly)
  • feat: generate tls certs for ui on helm install (#16601, @yandzee)
  • Fixes connectivity issues when kube-proxy replacement is enabled, caused by ineffective socket based load balancing (aka host reachable services) in the private cgroup namespace mode of container runtimes (e.g., docker cgroupv2 configuration). (#16259, @aditighag)
  • health: Add flag to set HTTP port (#16926, @errordeveloper)
  • helm: add back 'wellKnownIdentities' (#16142, @bmcustodio)
  • helm: Add support for disable-endpoint-crd option (#16226, @dntosas)
  • helm: Disable BPF masquerading in v1.10+ (#17824, @pchaigno)
  • helm: Disable the bandwidth manager by default (#16380, @pchaigno)
  • helm: Use batch/v1 apiVersion for CronJob in K8s 1.21+ (#16635, @gandro)
  • HTTP response access logs no longer contain the request headers, except for 'x-request-id', which is still included for request/response correlation purposes. (#16211, @jrajahalme)
  • Hubble logs for HTTP responses now include HTTP response headers. (#16013, @jrajahalme)
  • hubble/recorder: Extend the API to allow stopping a recording automatically (#16473, @gandro)
  • hubble: bump protoc{,-gen-go} and dependencies (#16915, @rolinh)
  • hubble: Hubble node_name field should contain cluster name (#15933, @Maddy007-maha)
  • images: Bump Hubble CLI to v0.8.0 (#15983, @gandro)
  • images: Bump Hubble CLI to v0.9.0 (Backport PR #18119, Upstream PR #18077, @gandro)
  • Improve Hubble memory usage and performance on decoding events (#17482, @tklauser)
  • install: Disable kube-proxy-replacement by default (#15422, @tgraf)
  • Make NodePort BPF to work on VLAN devices (#16772, @kvaster)
  • node-neigh: Locking, logging, misc improvements (#15783, @brb)
  • option: Rename egress gateway flag to enable-ipv4-egress-gateway (#17695, @pchaigno)
  • pkg/aws/eni: new subnet-ids parameter (#16119, @mvisonneau)
  • Pod L7 visibility annotations are now supported also when policy enforcement is enabled. (#16258, @jrajahalme)
  • Pod visibility annotations are now supported for Kafka and other policies implemented via Cilium Go extensions for Envoy. (#16935, @trvll)
  • Reduce bugtool memory usage (#17546, @tklauser)
  • Remove deprecated --update-ec2-apdater-limit-via-api option (#16374, @twpayne)
  • Remove deprecated code (#16502, @pchaigno)
  • Rename hostFirewall and mark stable (#17221, @pchaigno)
  • service: Always allocate higher ID for svc/backend (Backport PR #18119, Upstream PR #18113, @brb)
  • Skip iptables masquerading for packets destined to remote nodes (#16603, @pchaigno)
  • Store the previous Cilium's configuration options in the host (#16017, @aanm)
  • Support advertising Pod CIDRs via BGP (#16525, @christarazi)
  • Support EndpointSlices with BGP mode by updating MetalLB to v0.10.0 (#16524, @christarazi)
  • Support non-default Azure clouds (#16043, @ungureanuvladvictor)
  • Support TLS certificate auto-generation using certmanager (#17238, @dungdm93)
  • Use correct tolerations value when deploying cilium-operator via helm. (#15992, @michaelpetrov)
  • vendor: Update k8s dependencies and tests to 1.22.0-rc.0 (#16989, @nathanjsweet)
  • wireguard: Add fallback to userspace implementation (#17451, @gandro)
  • wireguard: Set wireguard and route MTU to detected MTU (#16020, @joamaki)

Bugfixes:

  • cluster-pool-ipv4-cidr and cluster-pool-ipv6-cidr options now accept string slices and not just string (#17780, @cndoit18)
  • Add '*.mesh.cilium.io' to the list of SANs for the server certificate of 'clustermesh-apiserver'. (#17027, @bmcustodio)
  • Adds an ACCEPT rule for untracked pkts in filter:CILIUM_OUTPUT (#17585, @Weil0ng)
  • Adds IPv6 support for generic-veth chaining plugin (#16041, @Weil0ng)
  • alibabacloud: fix race (#16175, @l1b0k)
  • bpf: exclude pod's reply traffic from egress gateway logic (#17869, @jibi)
  • bpf: fix hw_csum issue for icmp probe packets (#16604, @borkmann)
  • bpf: fix iptables masquerading for node -> remote pod traffic (#16136, @jibi)
  • bug/pkg/health: Fix Nil Address Issue in Node Update Mechanism (#17667, @nathanjsweet)
  • bugtool: fix data race occurring when running commands (#17916, @rolinh)
  • bugtool: fix IP route debug gathering commands (Backport PR #18076, Upstream PR #18059, @tklauser)
  • change log level for lock failed: endpoint is in the process of being removed (#16773, @humancalico)
  • Cilium Istio integration is updated to Istio release 1.10.4 (#17275, @jrajahalme)
  • cilium: Encryption EKS 4.14 kernel (default) fixes (#15867, @jrfastab)
  • daemon, node: Fix faulty router IP restoration logic (#16672, @christarazi)
  • daemon, node: Remove old, discarded router IPs from cilium_host (Backport PR #18076, Upstream PR #17762, @christarazi)
  • daemon: Ignore cilium_* interfaces when deriving NodePort device (#16104, @eyanulis)
  • daemon: require BPF masq to enable --install-no-conntrack-iptables-rules (#16085, @jibi)
  • datapath: Do not SNAT replies to outside (#17168, @brb)
  • datapath: panic explicitly when IP of direct-routing-device not found (#17064, @ArthurChiao)
  • datapath: Use TUNNEL_MODE as indicator for tunnel mode (#16328, @anfernee)
  • Define operator feature flags to allow the operator to register related CRDs. (#17772, @pchaigno)
  • DNS proxy is now more available during Cilium restarts, including upgrades. (#16391, @jrajahalme)
  • Drop a @ in clustermesh-apiserver helm chart (#15934, @anthr76)
  • egress gateway: fix non-tunnel (direct routing) mode (#17517, @kkourt)
  • egressgateway: Allow several CENPs with same egress IP (#17773, @pchaigno)
  • egressgateway: fix manager logic (Backport PR #18027, Upstream PR #17813, @jibi)
  • endpoint: trigger k8s sync controller on identity update (#16381, @jibi)
  • eni: Fix Cilium overallocating network interfaces (#15911, @gandro)
  • Envoy configuration is fixed to work also when IPv6 is disabled. (#17281, @rock-andy)
  • Envoy configuration with --proxy-prometheus-port is fixed. (#16834, @jrajahalme)
  • Envoy is updated to release 1.17.3 (#16102, @jrajahalme)
  • External Workloads service access is enabled again. (#16662, @jrajahalme)
  • Fix "unable to update ipcache map entry on pod add" harmless log warnings (#16286, @aanm)
  • Fix 5.10+ complexity issue with kubeProxyReplacement=disabled (#16084, @pchaigno)
  • Fix a crash where user specifies incorrect service name in a local redirect policy config, or policy selected service is added after the policy is added. (#16216, @aditighag)
  • Fix aws-cni integration where pods were not being scheduled (#15915, @aanm)
  • Fix bug where Cilium allocates a new router (cilium_host) IP upon node reboot, breaking connectivity especially with IPsec (#16307, @christarazi)
  • Fix bug where IP addresses of devices in unknown state are resolved as remote-node (#17418, @jibi)
  • Fix bug where L7 ingress policies with IPsec dropped traffic in tunneling mode (#16057, @christarazi)
  • Fix bug where the agents would silently skip all IPv6 masquerading due to an incorrect configuration. (#17906, @pchaigno)
  • Fix bug where timers used for retries sometimes fired immediately (#16955, @gandro)
  • Fix bug where users were unable to use node-selectors in the BGP configuration when using BGP support (#16341, @christarazi)
  • Fix bug with Helm chart where a user could not enable BGP and set Operator resources. (#16273, @rkage)
  • Fix identity leak via FQDN selectors (#17699, #17788, @joestringer)
  • Fix incorrect application of egress gateway policy to internal cluster traffic. Require a 5.2 kernel or later for the egress gateway policy feature. (#17639, @kkourt)
  • Fix incorrect packet path with IPsec and endpoint routes, which can cause incorrect policy drops. (#17000, @pchaigno)
  • Fix issue where generating Hubble certs were broken (#16509, @alex1989hu)
  • Fix issue where local host IPs may be briefly associated with the remote-node identity, causing policy drops when policy should allow traffic from the host. (#17836, @joestringer)
  • Fix Linux slave interface detection (#17189, @pchaigno)
  • Fix memory leak that can occur with the presence of FQDN policies (#17432, @aanm)
  • Fix several complexity and program size issues when only one of IPv4/IPv6 is enabled. (#17573, @pchaigno)
  • Fix transient policy deny during agent restart (#17115, @jaffcheng)
  • Fixed bug causing policy realization being skipped in some scenarios with endpoint identity churn. (#16271, @jrajahalme)
  • Fixes a bug where IPv6 pod CIDRs with leading zeros where not supported (#17707, @gandro)
  • Fixes an issue which can cause traffic to be dropped when running Cilium in ENI mode due to the presence of iptables rules left over by the AWS VPC CNI plugin. Notable features that could be impacted include the egress gateway functionality. (#17845, @bmcustodio)
  • Fixes for IPsec and endpoint routes (#17865, @kkourt)
  • Fixes out-of-sycn CEP update (#17001, @Weil0ng)
  • helm: Fix operator cloud image digests (Backport PR #18119, Upstream PR #18116, @joestringer)
  • helm: Fix patch failure when updating hubble-generate-certs (#16373, @gandro)
  • helm: upgrade envoy to v1.18.4 for hubble-ui (#17439, @geakstr)
  • hubble/recorder: Refactor service implementation to fix multiple races (#16472, @gandro)
  • hubble: Display proxy redirects in policy verdict events (#17411, @pchaigno)
  • hubble: Never fail with ErrInvalidRead (#17046, @michi-covalent)
  • Ignore K8s namespace events that have the same labels (#16268, @aanm)
  • install: Allow setting enable-health-check-nodeport to 'false' (#16323, @dctrwatson)
  • ipam/crd: Fix spurious "Unable to update CiliumNode custom resource" failures in cilium-agent (Backport PR #18027, Upstream PR #17856, @gandro)
  • ipam: fix crd mode (#16493, @joamaki)
  • ipsec: Fix logging of SPI after key rotations (#16557, @pchaigno)
  • ipsec: Fix off-by-one error on max keyID (#16647, @pchaigno)
  • iptables: Remove leading zeroes (#16817, @jrajahalme)
  • L7 proxy redirection on IPv6 ingress to a pod is fixed to properly update IPv6 hop limit. (#17718, @jrajahalme)
  • lbmap: fix deletion and recreation logic for maglev maps (#16850, @jibi)
  • loader: Revert incorrect initialization of endpoints in chaining mode (#16227, @pchaigno)
  • lrp: Skip clusterIP service restore in service delete callback (#16548, @aditighag)
  • node-init: cleanup snat iptables rules when running in eni mode with masquerading disabled (#16840, @bmcustodio)
  • node: Fix race condition on labels' getter/setter (#17217, @pchaigno)
  • node: Skip ipcache for remote node IPs if IPsec is enabled (#17511, @pchaigno)
  • Operator gc incluster identities only (#17589, @ArthurChiao)
  • operator: only GC identity keys of its own cluster (#16825, @ArthurChiao)
  • Optimize memory consumption for clusters with high number of repeated FQDN matchPattern or matchNames (#17224, @aanm)
  • Perform reverse NAT at host interface (#15354, @krishgobinath)
  • pkg/identity: Add missing labels to well-known identities (#16585, @mauriciovasquezbernal)
  • pkg/k8s: fix invalid memory address or nil pointer dereference (#17642, @aanm)
  • pkg/option: Fix default assignment of EnableWellKnownIdentities (#16434, @mauriciovasquezbernal)
  • Plumb Azure interface's VPC / primary CIDR and set it as native routing CIDR in Azure IPAM mode (#16696, @christarazi)
  • policy: Fix cilium policy trace output when only deny rules are applied (#16991, @chez-shanpu)
  • Potential deadlock in pod identity updates has been fixed. (#16529, #16801, @jrajahalme)
  • Prometheus lint errors in operator metrics (Backport PR #18076, Upstream PR #17789, @krishgobinath)
  • Remove node.cilium.io/agent-not-ready node taints if they are re-added after Cilium has started (#17112, @aanm)
  • Remove CiliumNode deletion logic from CiliumNode watcher and guarantee CiliumNode's OwnerReference is always set (#17329, @christarazi)
  • Remove previous PERM ARP entries installed by Cilium when kube-proxy-replacement and IPSec are disabled. (#16359, @aanm)
  • Removes cilium daemonset's dependencies on utilities like sh and mount having installed in the underlying host distributions. (#16815, @aditighag)
  • routing: Fix incorrect interface selection for egress pod routes (#17169, @pchaigno)
  • Set right User Agent in Kubernetes client for all Cilium components. (#17417, @aanm)
  • ui envoy: fix config to keep grpc conn (#15938, @geakstr)
  • wireguard: Fix traffic counters in cilium debuginfo (#16178, @gandro)

CI Changes:

Misc Changes:

Docker Manifests

cilium

docker.io/cilium/cilium:v1.11.0@sha256:ea677508010800214b0b5497055f38ed3bff57963fa2399bcb1c69cf9476453a
quay.io/cilium/cilium:v1.11.0@sha256:ea677508010800214b0b5497055f38ed3bff57963fa2399bcb1c69cf9476453a

clustermesh-apiserver

docker.io/cilium/clustermesh-apiserver:v1.11.0@sha256:361942671ce067cc7f3e97c2114512283148bcee5ec29e4f0a828869aedd4ced
quay.io/cilium/clustermesh-apiserver:v1.11.0@sha256:361942671ce067cc7f3e97c2114512283148bcee5ec29e4f0a828869aedd4ced

docker-plugin

docker.io/cilium/docker-plugin:v1.11.0@sha256:2b7df46918ba832f7c55bc7255f8599af30aa8dc43d62f854b7f10b43f8387c9
quay.io/cilium/docker-plugin:v1.11.0@sha256:2b7df46918ba832f7c55bc7255f8599af30aa8dc43d62f854b7f10b43f8387c9

hubble-relay

docker.io/cilium/hubble-relay:v1.11.0@sha256:306ce38354a0a892b0c175ae7013cf178a46b79f51c52adb5465d87f14df0838
quay.io/cilium/hubble-relay:v1.11.0@sha256:306ce38354a0a892b0c175ae7013cf178a46b79f51c52adb5465d87f14df0838

operator-alibabacloud

docker.io/cilium/operator-alibabacloud:v1.11.0@sha256:e61929869d59c5093c6d129ca1c21386338e1387051779d499a988545680b00a
quay.io/cilium/operator-alibabacloud:v1.11.0@sha256:e61929869d59c5093c6d129ca1c21386338e1387051779d499a988545680b00a

operator-aws

docker.io/cilium/operator-aws:v1.11.0@sha256:5f60a4e17ab33a3dcd2a942802b15f9e7be3d18f24464f31bba81a65a117e094
quay.io/cilium/operator-aws:v1.11.0@sha256:5f60a4e17ab33a3dcd2a942802b15f9e7be3d18f24464f31bba81a65a117e094

operator-azure

docker.io/cilium/operator-azure:v1.11.0@sha256:c1b41e6cbf6f1e0bb417170ac79eb6d78a7e39b775f1131a1104546fd18d745f
quay.io/cilium/operator-azure:v1.11.0@sha256:c1b41e6cbf6f1e0bb417170ac79eb6d78a7e39b775f1131a1104546fd18d745f

operator-generic

docker.io/cilium/operator-generic:v1.11.0@sha256:b522279577d0d5f1ad7cadaacb7321d1b172d8ae8c8bc816e503c897b420cfe3
quay.io/cilium/operator-generic:v1.11.0@sha256:b522279577d0d5f1ad7cadaacb7321d1b172d8ae8c8bc816e503c897b420cfe3

operator

docker.io/cilium/operator:v1.11.0@sha256:c802c16b7ab561075c08779c0e4c53acdb97753c38f27424bc243e444aa524b9
quay.io/cilium/operator:v1.11.0@sha256:c802c16b7ab561075c08779c0e4c53acdb97753c38f27424bc243e444aa524b9

Contributors

rkage, twpayne, and 110 other contributors
Assets 2