[EKS] [request]: Allow specifying which admission controllers are enabled

[hendrikhalkow]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
When I create an EKS cluster, I'd like to specify which of the compiled-in admission controllers are enabled.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
According to the documentation, there is an immutable list of admission controllers that are enabled by default for each EKS version. I'd like to use the AlwaysPullImages admission controller without using dynamic admission control.

Are you currently working around this issue?
Since the AlwaysPullImages is the only admission controller for which I'd like to have this, I use immutable image tags.

[brychcy]

It should be possible to enable the ExtendedResourceToleration admission controller, which is according to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#example-use-cases the recommended way to restrict GPU nodes to pods that need them.

[SQUIDwarrior]

Yes this is essential to allowing Spark users to schedule GPUs on EKS since Spark 2.4 does not provide a way to modify the Kubernetes operations manually.

[shinji62]

@mikestef9 anything we can do to bump this issue ?

[mikestef9]

We will be enabling the ExtendedResourceToleration admission controller with the upcoming 1.19 launch. Are there any other admission controllers you are interested in?

[Elgarni]

@mikestef9 I would also like to see PodNodeSelector enabled, to be able to select nodes on the namespace level,as described in k8s docs here

[evanstucker-hates-2fa]

I want the AlwaysPullImages controller so our multi-tenant dev clusters are more secure, and so I don't have to add imagePullPolicy: Always to everything everywhere - it will just be Always by default. https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages

[joshpaul-okta]

We are interested in the following two admission controllers as used together allow for explicit inherited and enforced placement of workloads at the namespace level.

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction

Adding the labels to the namespace the restricts all resources in the namespace to the target nodegroup and will cause them to be inherited so teams do not have to work about this. With the additional use of taints on the nodegroups we get guarantees on workloads not being able to drift from the targeted capacity.

[ryanisnan]

Our use-case is similar to @joshpaul-okta.

We'd really like to be able to use the PodTolerationRestriction admission controller. We have a managed node-group which we'd like to reserve for a wide-number of community-provided helm chart resources. The helm charts that we're including can be really complex, and having to add tolerations manually is tedious, error prone, and not very resilient to change.

Being able to add tolerations to pods by namespace configuration would make this go from being a real nightmare of a problem, to something I never think about.

[atamgp]

@mikestef9
What are the plans for podtolerationrestriction?
Platform is already on 1.24 but no mention of this.

[evanstucker-hates-2fa]

imagePullPolicy: Always means that every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container (https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy).

This is desirable for security for a few reasons:

• "Relying on cached versions of a Docker image can become a security vulnerability. By default, an image will be pulled if it isn't already cached on the node attempting to run it. This can result in variations in images that are running per node, or potentially provide a way to gain access to an image without having direct access to the ImagePullSecret." (https://www.fairwinds.com/blog/kubernetes-how-to-ensure-imagepullpolicy-set-to-always)

• "Admin1 creates Pod1 that uses SuperSecretImage1 by specifying ImagePullSecret1 to gain access to the SecureContainerRegistry holding the image. Kubernetes pulls down and caches SuperSecretImage1 on the node and then creates Pod1 accordingly. Admin2, who does not have access to ImagePullSecret1 or SecureContainerRegistry, attempts to create Pod2 using SuperSecretImage1 and an imagePullPolicy of IfNotPresent. Because this image is cached on the node, this operation is successful." (https://trstringer.com/kubernetes-alwayspullimages/) The mitigation for this is to use an AdmissionController that forces ImagePullPolicy to Always on all workloads.

We've been waiting for this for almost 3 years now. I suppose it’s not as critical for me in particular now that we have separate clusters per team... And it’s definitely more profitable for AWS to push people towards a cluster-per-team paradigm, instead of making it easier to have efficient, secure multi-tenant clusters.

Are we ever going to see this feature added?

[daniel-laszlo]

Another usecase is enforcing Pod Security Standards compliance clusterwide: https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/

[joebowbeer]

The admission controllers mentioned in the comments are

• AlwaysPullImages
• ExtendedResourceToleration
• PodNodeSelector - still alpha since 1.5
• PodTolerationRestriction - still alpha since 1.7

I wonder how many of these requests can be resolved by the ValidatingAdmissionPolicy that is alpha in 1.26?

In addition, there is a request to support configuring the PodSecurity admission controller that is enabled by default in 1.25. This should be a separate issue?

Relates to #512

[dims]

Here's something interesting in 1.28 when upstream kubernetes ships it .. CEL for Admission Control

• The overall issue for this feature is CEL for Admission Control kubernetes/enhancements#3488
• The KEP itself is https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/3488-cel-admission-control

Bringing this up since some of those admission controllers stuck in alpha for a VERY long time won't make it into beta. So CEL is the new way to give folks a way to do what they would like to do without touching the API server itself. See kubernetes/kubernetes#117837 on a PR to deprecate and eventually remove PodNodeSelector and PodTolerationRestriction. That PR will land when kubernetes master branch opens up for 1.29 work.