Repository navigation
Support removing ephemeral container from pod #84764
Description
What would you like to be added:
Ephemeral container can be removed from pod gracefully.
Why is this needed:
In our K8s cluster, there are many deployed pods. We aim to use ephemeral containers as troubleshooting containers. Thus, for troubleshooting, many ephemeral containers are potentially created.
Once the need of an ephemeral container is gone, the container should be removed gracefully. Resource of the container should be released. The container should not be accessed any more for security. And the container also should be removed from pod.
Unless the pod which contains ephemeral container is destroyed, there are no ways to remove the ephemeral container totally.
However, this approach is really "intrusive". (For example, traffic may be impacted) We need a way to remove ephemeral container gracefully like how we creating it.
Activity
shuiqing05 commentedon Nov 5, 2019
/assign @verb
verb commentedon Nov 5, 2019
/sig node
Thanks @shuiqing05 for opening this issue! Is your concern mainly about reclaiming the resources once the ephemeral container has exited? This is a thing that should happen automatically regardless of whether the container has been removed from the PodSpec. It shouldn't require a manual step. (To be clear, I'm not saying this is current behavior.)
Deleting the container would be required if, for example, you wanted to reclaim the resources without the container having exited.
dguendisch commentedon Nov 7, 2019
My usecases for removing closely match @shuiqing05 ones:
shareProcessNamespace: true); after deploying an allmighty ephemeral debug container I want to remove it after debugging was done so that my minimal container stays minimal againfejta-bot commentedon Feb 5, 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
tedyu commentedon Feb 5, 2020
@dguendisch @shuiqing05
What would be the desired policy for removing ephemeral containers ?
For security purpose, should the container be denied access to the Pod after configurable duration ?
dguendisch commentedon Feb 5, 2020
For my usecases I wouldn't need duration based policies. My main point would be to be able to manually delete/remove the eph container (after I'm done debugging the problem). Nice to have would be the option/policy of having it automatically removed after the container exited.
duglin commentedon Feb 7, 2020
Hi - what's the status of this? I'm really interested in using ephemeral containers for some of my usecases but, like others, I'd like to be able to remove them and reclaim all resources as the pod itself might live for a long time but these e-containers would come and go quite frequently.
verb commentedon Feb 10, 2020
@dguendisch @tedyu Rather than functionality in the kubelet to automatically remove a container based on a policy enforced by the kubelet, I prefer marking a pod as tainted when an ephemeral container has been added (#84353) and leaving it up the administrator how to handle this situation. I could imagine a controller than implements an automatic removal policy.
@duglin It hasn't been designed yet. Right now I'm focused on a debugging MVP. If anyone wants to pick this up the next step would be to gather requirements and propose an update to the KEP.
verb commentedon Mar 5, 2020
I'm considering how to do this for 1.19
/remove-lifecycle stale
38 remaining items
tonyaw commentedon May 12, 2022
@verb and @pacoxu,
Could you please help to provide the latest status to support removing ephemeral container?
I'd like to use ephemeral container to do some periodic check, and my concern is if the pod spec can be increased more than 1.5 MiB which is the limit of etcd.
pacoxu commentedon May 12, 2022
The KEP was updated in the v1.23 release cycle kubernetes/enhancements#2892.
There are some discussions in #103354 (comment).
tonyaw commentedon May 12, 2022
Can I say I can't use ephemeral container to run a periodic check for a long-lived pods by using current release?
I need the ephemeral container to run and exit after each check.
pacoxu commentedon May 12, 2022
No, you can run the periodic check. (The pod's ephemeral containers would be +1 each time and ephemeral containers would be too many after a long period.)
The Pod can have many ephemeral containers with different names. This issue is about removing exited containers from pod's ephemeral container list. Once the ephemeral container was added to the Pod, it cannot be removed.
tonyaw commentedon May 12, 2022
Right. My concern is if ephemeral containers run too many times after a long period, the list will be too long and make pod manifest reach 1.5MiB etcd limiation.
Will it be a problem? :-)
liggitt commentedon May 12, 2022
Using ephemeral containers to run a planned periodic check indefinitely for a long-running pod will monotonically increase the length of the ephemeral containers list, and is not a good use case for this feature. You could include what you need to run the periodic check in one of the normal containers and use something like
execto run the checktonyaw commentedon May 12, 2022
Thanks @liggitt.
The problem here is:
Our application container is delivered without any debug tools, such as ping, and I want to use ephemeral container to try a periodic ping check to ensure the network is good.
It shall be aligned with one benifit of ephermal container mentioned here:
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/#uses-for-ephemeral-containers
"In particular, distroless images enable you to deploy minimal container images that reduce attack surface and exposure to bugs and vulnerabilities. Since distroless images do not include a shell or any debugging utilities, it's difficult to troubleshoot distroless images using kubectl exec alone."
But as you mentioned, the ephemeral containers list will monotonically increase the length, so removing ephermal container is a mandatory feature for my use case.
If there is any other way to do it, could you please help to provide it?
liggitt commentedon May 12, 2022
What you're describing sounds more like a health or liveness check, not one-off debugging... including what you need to run that in the normal containers for the app is required.
tonyaw commentedon May 12, 2022
Based on these two reasons, we can't run ping inside application container.
ephemeral container is a perfect match since it not only can use different container image comparing with application container, but also get default capability including NEW_RAW.
The only missing is removing ephermal container feature. :-)
As it also menitoned in following link, ephermal container is designed for on-off debugging at beggining, but there is indeed some needs for repeat run:
allow removing ephemeral containers since 1.23 #103354 (comment)
Could you please help to provide the reason for why not to add removing ephemeral container feature? Is ephermal container still only designed for one-off debugging purpose?
liggitt commentedon May 12, 2022
The reason remove was not supported was for auditability (losing API-visible information about what has been added to the pod and run) and to avoid race conditions where an ephemeral container with the same name was removed and re-added and appeared to the kubelet to change in-place.
tonyaw commentedon May 12, 2022
@liggitt Thanks for your info.
From KEP, looks like there is no plan to add the removing feature in the short term. Right?
Also, for the use case I mentioned, I think there is no solution so far. Right? :-(
verb commentedon Jun 10, 2022
@tonyaw That's correct, we won't be implementing this in the context of kubernetes/enhancements#277. It's possible we'd revisit this in the context of a new KEP, since it's popular, but I wouldn't expect it in the short term.
I agree that your use case sounds like a bad fit for ephemeral containers, which are best effort and shouldn't be part of normal pod operation. You could implement this as a health check that doesn't use ICMP (and so doesn't require NET_RAW) but I'd be worried about all of the pods exiting at once if the destination host became unavailable for some reason.
unmarshall commentedon Aug 18, 2022
If you need for audit reasons then why bloat the pod spec and status? Instead create yet another companion resource which has audit information (gets created when the first debug/attach is issued). This way you do not have to change the pod spec, also allows you to control how much audit information needs to be kept which could also be made configurable. Operators/Consumers can also switch on/off auditing for selected resources which they control etc.
Spec is used to describe the desired state and should not be polluted with audit trail information which can keep growing. Auditing is a cross cutting concern. A desired state should not have ever-growing audit information as part of it. IMHO it simply does not belong there.
jackmtpt commentedon Nov 17, 2023
Is there any update on this? Any plans to allow deleting ephemeral containers?
chengjoey commentedon Apr 16, 2024
#124271
draft for removing ephemeral container from pod