Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
rkt: Support per-pod stage1-image #23944
It would be useful to be able to set the rkt stage1-image on a per-pod basis (overriding the global
A example use-case would be using the rkt-fly stage1 when running a self-hosted kubelet, while using the global stage1 for all other pods.
One option might be to use annotations on the pod which the rkt runtime could parse for stage1-image. This would be somewhat lightweight in that no api-changes would need to be made. Eventually this could be replaced if/when runtime-specific pod configuration was available.
A concern with using annotations is that the stage1 could essentially be running the pod with escalated privileges. It might mean that at the kubelet side, pods with a stage1 annotation would need to be rejected if they did not also contain the
To add a bit more detail to this request:
From a selfish "I want this" perspective:
This comes from the need to be able to run a kubelet pod using specific rkt runtime options (in this case, using a different stage1-image). This would currently be necessary for running a self-hosted kubelet with rkt as the container runtime.
Starting from a self-hosted kubelet proposal: #23343
From a more general perspective:
There might be some other options that would allow a self-hosted kubelet with rkt as the container runtime, potentially without needing to specify per-pod runtime options.
However, it seems like container runtime configuration (potentially at the pod level) is going to be somewhat necessary no matter what. Some of the options in the pod spec are already runtime specific, and may not map well to various container runtimes. For example, what does "hostPID" mean when the runtime could be VM based (e.g. hyper or kvm isolation)? Or, how/why is some runtime configuration blessed, but potentially not others?
So I guess back to the originally posted (and selfish) issue: being able to specify a runtime configurable (stage1-image in this case) as part of the pod would allow for situations where we may need to change the isolation mechanics of a pod (in a concrete use-case: for self-hosting the kubelet).
@yifan-gu @philips For now, experimental runtime-specific options could be passed via annotations on the pod. Something like
referenced this issue
Apr 21, 2016
I am happy to implementing something like this.
@bgrant0607 But the problem for using annotation for stage1 image in rkt, as mentioned in #23944 (comment), is that this could give users privilege without noticing the API server, (users can run arbitrary stage1, unless we verify this in kubelet/rkt) and API server has no way to check and prevent it..
And if we let the kubelet fail those pods with illegal stage1 image annotation, then we probable see a crash loop that keeps retrying... Though that might not be too bad for now?
If the crashloop / events are clear that the reason a pod is failing to run is that a