CiliumEndpointSlice Graduation to Stable

[thorn3r]

CiliumEndpointSlice Graduation to Stable

SIG: sig-scalability

Begin Design Discussion: 2024-04-11

Cilium Release: 1.16

Authors: Tim Horner timothy.horner@isovalent.com, Marcel Zięba marcel.zieba@isovalent.com, Marco Iorio marco.iorio@isovalent.com

Summary

CiliumEndpointSlice facilitates the grouping of multiple CiliumEndpoint objects to achieve better scalability. This document describes the remaining work needed to consider CiliumEndpointSlice a Stable feature.

Goals

• Blanket test CiliumEndpointSlice in all CI workflows.

  • CiliumEndpointSlice will be temporarily enabled for all CI workflows to identify failures or feature incompatibilities. The goal of the blanket test is to provide guidance on areas of improvement to focus on and any gaps in tooling.

• Enable CiliumEndpointSlice in CI tests.

  • The feature has unit test coverage but there is no end-to-end testing. CiliumEndpointSlice will be enabled in the Conformance E2E and 100 Nodes Scale Test workflows. Additional workflows may be added based on the findings of the blanket test.

• Identify feature incompatibilites

  • Identify features that rely on the use of CiliumEndpoint and summarize the work needed to support CiliumEndpointSlice. A decision should be made per-feature whether the incompatibility should be fixed or documented. The decision should be made based on the amount of work needed and the value add of making the feature compatible with CiliumEndpointSlice.

• Simplify CiliumEndpointSlice configuration

  • Cilium Operator accepts the following 8 options for configuring CiliumEndpointSlice:

    • ces-max-ciliumendpoints-per-ces
    • ces-slice-mode
    • ces-write-qps-limit
    • ces-write-qps-burst
    • ces-enable-dynamic-rate-limit
    • ces-dynamic-rate-limit-nodes
    • ces-dynamic-rate-limit-qps-limit
    • ces-dynamic-rate-limit-qps-burst

  • Configuration should be simplified by setting sane defaults and removing any unused or deprecated options. CiliumEndpointSlice should use its own Kubernetes ClientSet to decouple it from the ClientSet used by the CiliumOperator for all other resources.

• Define a migration path from CiliumEndpoint to CiliumEndpointSlice

  • A migration path is needed to allow users to enable CiliumEndpointSlice on clusters utilizing CiliumEndpoint. The process should be simple and well documented, with CI test coverage to identify regressions.
  • CI coverage for migration path from CiliumEndpoint to CiliumEndpointSlice #32057

Tasks

Beta Give feedback

1. Blanket test CiliumEndpointSlice in all CI workflows #32589
   +0
   
2. Enable CiliumEndpointSlice in CI tests #32590
   +0
   
3. EgressGW: support CiliumEndpointSlices #24833
   feature/egress-gateway kind/enhancement pinned sig/datapath +4
4. Simplify CiliumEndpointSlice configuration #32591
   +0
   
5. CI coverage for migration path from CiliumEndpoint to CiliumEndpointSlice #32057
   area/CI area/CI-improvement sig/agent +3
   
6. CODEOWNERS: add sig-scalability ownership of CiliumEndpointSlice #32535
   ready-to-merge release-note/misc +2
   

[antonipp]

Thank you for starting this! 🙌

Just for future reference, my PR #31800 already has the docs for the Define a migration path from CiliumEndpoint to CiliumEndpointSlice item (although there are no tests)

[thorn3r]

@antonipp I saw that, nice work and thank you for writing that up!

[marseel]

Issue related to CES and ces-slice-mode option: #31564

[dlapcevic]

I think ces-slice-mode should be deprecated together with Identity batching mode, and FCFS batching mode will be the default and only one.

Reason:
FCFS is a simpler algorithm (fewer possible bugs) that has better performance for almost all cases.

[dlapcevic]

Is the purpose of having a separate K8s client for CES to ensure that it doesn't impact or get impacted by other resources -- client side rate limiting for example?

If so, is there a pattern that can be applied across resources? Which resources require a separate client?

[marseel]

If so, is there a pattern that can be applied across resources? Which resources require a separate client?

I don't think there is a pattern in Cilium yet, but definitely, there are other controllers for which we would like to do it too.
The pattern is mostly inspired by kcm in k8s.

[jshr-w]

I would like to help with the CI coverage for the migration path, as in the issue above. Would someone be able to help assign the issue to me, please?

[thorn3r]

Agreed that the identity-based mode should be deprecated. We should also add some validation for the identity slice mode option. Currently, if anything other than cesSliceModeIdentity is supplied (including typos), you get FCFS. The risk is low, but still worth covering the bases imo.