Sidecar Containers

[Joseph-Irving]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes introduces the built-in support for sidecar containers pattern.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/753-sidecar-containers/README.md
• Discussion Link: Sidecar working group and KEP draft
• Primary contact (assignee): @SergeyKanzhelev
• Responsible SIGs: node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): TBD
• Alpha
  • KEP (k/enhancements) update PR(s):
    • Sidecar containers KEP #3761
    • move sidecar to milestone 1.28 #3968
  • Code (k/k) update PR(s):
    • Add SidecarContainers feature kubernetes#116429
  • Docs (k/website) update PR(s):
    • Add initial (alpha) documentation about sidecar containers website#41802
• Beta
  • KEP (k/enhancements) update PR(s): KEP-753: add PRR questionnaire answers for beta #4255
  • Code (k/k) update PR(s):
    • sidecars: terminate sidecars after main containers kubernetes#120620
    • Promote SidecarContainers feature to beta kubernetes#121579
    • Add cel new validator into Kubernetes kubernetes#121307
  • Docs (k/website) update(s): Document SidecarContainers graduation to beta website#43471

/sig node

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[Joseph-Irving]

@enisoc @dchen1107 @fejta @thockin @kow3ns @derekwaynecarr, opened this tracking issue so that we can discuss.

[kow3ns]

/assign

[Joseph-Irving]

@derekwaynecarr I've done some scoping out of the kubelet changes required for next week's sig-node meeting, I believe that changes are only needed in the kuberuntime package, specifically kuberuntime_manager.go in and kuberuntime_container.go.

In kuberuntime_manager.go you could modify computePodActions to implement the shutdown triggering (kill sidecars when all non-sidecars have permanently exited), and starting up the sidecars first.

In kuberuntime_container.go you could modify killContainersWithSyncResult for terminating the sidecars last and sending the preStop hooks (the preStop hooks bit was a bit debatable, it wasn't settled whether that should be done or not. @thockin had a good point about why you might not want to encourage that behaviour, see comment).

Let me know if you want me to investigate any further.

[resouer]

@kow3ns The discussion makes more sense to me if maybe we can define a full description of containers sequence in Pod spec (sig-app), and how to handle the sequence in kubelet for start, restart and cascading consideration (sig-node). Let's catch the Feb 5 sig-node meeting to give more inputs.

cc @Joseph-Irving

[luksa]

The proposal says that sidecars only run after the init containers run. But what if the use-case requires the sidecar to run while/before the init containers run. For example, if you'd like route the pod's traffic through a proxy running as a sidecar (as in Istio), you probably want that proxy to be in place while the init containers run in case the init container itself does network calls.

[Joseph-Irving]

@luksa I think there's the possibility of looking at having sidecars that run in init phase at some point but currently the proposal is not going to cover that use case. There is currently no way to have concurrent containers running in the init phase so that would be potentially a much larger/messier change than what is being suggested here.

[Joseph-Irving]

Update on this KEP:
I've spoken to both @derekwaynecarr and @dchen1107 from sig-node about this and they did not express any major concerns about the proposal. I will raise a PR to the KEP adding some initial notes around implementation details and clarifying a few points that came up during the discussion.

We still need to agree on the API, it seems there is consensus that a simple way of marking containers as sidecars is prefered over more in depth ordering flags. Having a bool is somewhat limiting though so perhaps something more along the lines of containerLifecycle: Sidecar would be preferable so that we have the option of expanding in the future.

[luksa]

@Joseph-Irving Actually, neither the boolean nor the containerLifecycle: Sidecar are appropriate for proper future extensibility. Instead, containerLifecycle should be an object, just like deployment.spec.strategy, with type: Sidecar. This would allow us to then introduce additional fields. For the "sidecar for the whole lifetime of the pod" solution, it would be expressed along these lines:

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: CompletePodLifetime

as opposed to

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: AfterInit

Please forgive my bad naming - I hope the names convey the idea.

But there is one problem with the approach where we introduce containerLifecycle to pod.spec.containers. Namely, it's wrong to have sidecars that run parallel to init containers specified under pod.spec.containers. So if you really want to be able to extend this to init containers eventually, you should find an alternative solution - one that would allow you to mark containers as sidecars at a higher level - i.e. not under pod.spec.containers or pod.spec.initContainers, but something like pod.spec.sidecarContainers, which I believe you already discussed, but dismissed. The init containers problem definitely calls for a solution along these lines.

[Joseph-Irving]

@luksa You could also solve the init problem by just allowing an init container to be marked as a sidecar and have that run alongside the init containers. As I understand it, the problem is that init containers sometimes need sidecars, which is different from needing a container that runs for the entire lifetime of the pod.

The problem with pod.spec.sidecarContainers is that it's a far more complex change, tooling would need to updated and the kubelet would require a lot of modifying to support another set of containers. The current proposal is far more modest, it's only building on what's already there.

[luksa]

@Joseph-Irving We could work with that yes. It's not ideal for the sidecar to shut down after the init containers run and then have the same sidecar start up again, but it's better than not having that option. The bigger problem is that older Kubelets wouldn't handle init-sidecar containers properly (as is the case with main-sidecar containers).

I'd just like you to keep init-sidecars in mind when finalizing the proposal. In essence, you're introducing the concept of "sidecar" into k8s (previously, we basically only had a set of containers that were all equal). Now you're introducing actual sidecars, so IMHO, you really should think this out thoroughly and not dismiss a very important sidecar use-case.

I'd be happy to help with implementing this. Without it, Istio can't provide its features to init containers (actually, in a properly secured Kubernetes cluster running Istio, init containers completely lose the ability to talk to any service).

[Joseph-Irving]

In relation to the implementation discussion in #841, I've opened a WIP PR containing a basic PoC for this proposal kubernetes/kubernetes#75099. It's just a first draft and obviously not perfect but the basic functionality works and gives you an idea of the amount of change required.

cc @enisoc

[Joseph-Irving]

I put together a short video just showing how the PoC currently behaves https://youtu.be/4hC8t6_8bTs. Seeing it in action can be better than reading about it.
Disclaimer: I'm not a pro youtuber.

[Joseph-Irving]

I've opened two new PRs:

• add release checklist, grad criteria, upgrade strategy to sidecar kep #918 is about Graduation critieria, version skew, upgrade/downgrade
• Sidecar KEP API implementation #919 is to decide on the API implementation

Any thoughts or suggestions will be much appreciated.

[SergeyKanzhelev]

/milestone v1.29
/label lead-opted-in

We will be making an update for termination ordering. We want to start conversation if this will be OK for beta given the huge interest in a community

[sanchita-07]

Hello @SergeyKanzhelev 👋, 1.29 Enhancements team here!

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage beta for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, if it is targeting beta for v1.29, we need to have the above milestones completed.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

Thank you!

[sanchita-07]

Hi @SergeyKanzhelev , checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk for enhancement freeze. It looks like we need to have the above milestones completed if it is targeting beta for v1.29. Let me know if I missed anything. Thanks!

[SergeyKanzhelev]

/stage beta

got approved for beta in 1.29!!!

[Priyankasaggu11929]

PRs #4255 and #4183 addressing the 1.29 Enhancements Freeze criteria, are merged.

[npolshakova]

The status is now marked as tracked for enhancement freeze for 1.29. 🚀 Thank you!

[drewhagen]

Hello @SergeyKanzhelev 👋, v1.29 Docs Shadow here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[matthyx]

@drewhagen kubernetes/website#43471

[a-mccarthy]

Hi @SergeyKanzhelev, 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository.
The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here is the 1.29 calendar

[sanchita-07]

Hey again @SergeyKanzhelev 👋, 1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023:

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Please update the Issue description to include all the related PRs of this KEP under a new beta section in the Github Issue description. The status for this KEP is currently at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.

As always, we are here to help if any questions come up ✌. Thanks :)

[npolshakova]

Hello @SergeyKanzhelev 👋, 1.29 Enhancements team here!

With all the implementation(code related) PRs merged as per the issue description, this enhancement is now marked as tracked for code freeze for the 1.29 Code Freeze! 🚀

Please update the issue description with the merged PRs:

• sidecars: terminate sidecars after main containers kubernetes#120620
• Promote SidecarContainers feature to beta kubernetes#121579
• Add cel new validator into Kubernetes kubernetes#121307

Let me know if I missed any additional code PRs that were merged as part of the beta stage for 1.29.

The test freeze is 01:00 UTC Wednesday 15th November 2023 / 18:00 PDT Tuesday 14th November 2023. Please make sure all test PRs are merged in by then. Let me know if there are additional test PRs we should track. Thanks!

[salehsedghpour]

/remove-label lead-opted-in

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[SergeyKanzhelev]

this KEP will not progress in 1.30 and will stay in beta. As per per https://kubernetes.slack.com/archives/CPNHUMN74/p1706017979795519 @matthyx will fork out another KEP for improved Pods termination logic.

@matthyx pls ping me when done so I can tag the forked KEP for 1.30 milestone

[pacoxu]

@matthyx pls ping me when done so I can tag the forked KEP for 1.30 milestone

#4324 is updated using a seprate KEP.

And the new KEP issue is #4438.