Add SidecarContainers feature

[gjkim42]

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This adds the Sidecar Containers feature to kubernetes.

• API change - add restartPolicy to Init containers
• API validation - only allow restartPolicy for Init containers and allow startupProbe for the sidecar containers
• Startup: wait for Started, not for completion.
• Reconciling status of Sidecar container with the runtime
• Termination: terminate the pod even if sidecar is still running
• Resources calculation update
• Restart sidecar on sidecar failure/completion while pod is still running
• Topology/CPU managers update

Which issue(s) this PR fixes:

Fixes #

ref: #115934

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers

/cc @SergeyKanzhelev @matthyx

[cici37]

/cc @jpbetz

[matthyx]

/triage accepted
/priority important-soon

[jpbetz]

Here's some API review related feedback, I'll do a pass on the implementation shortly

[SergeyKanzhelev]

I updated the state of this PR in this issue description: #115934

There are still some things to be done, but those are tracked and close to be implemented

[olitomlinson]

The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.

Quick one, just to be more explicit, would the sidecar init container start before other non-sidecar init containers? or would it start in parallel to non-sidecar init containers? TIA

Apologies if this is the wrong place to ask the question, I'm new here :)

[matthyx]

The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.

Quick one, just to be more explicit, would the sidecar init container start before other non-sidecar init containers? or would it start in parallel to non-sidecar init containers? TIA

Apologies if this is the wrong place to ask the question, I'm new here :)

It depends on which order you put them inside initContainers... there is only one ordered list of init containers, some of which are executed to completion (non-sidecars) and the new ones launched and restarted if needed (sidecars).

[tzneal]

Quick one, just to be more explicit, would the sidecar init container start before other non-sidecar init containers? or would it start in parallel to non-sidecar init containers? TIA

The sidecar init containers follow the standard startup order logic (i.e. in the order specified).

[matthyx]

The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.

Quick one, just to be more explicit, would the sidecar init container start before other non-sidecar init containers? or would it start in parallel to non-sidecar init containers? TIA
Apologies if this is the wrong place to ask the question, I'm new here :)

It depends on which order you put them inside initContainers... there is only one ordered list of init containers, some of which are executed to completion (non-sidecars) and the new ones launched and restarted if needed (sidecars).

I need to finish the docs PR and make it clear there before we release...