Make it possible to shell into any container, even if it has no shell

Today kubectl exec sh depends on a shell being present in each container image. I think we can do better.

What if we had something like kubectl sh which would run a statically linked busybox (or something like that) that was provided by the host node and made that enter the cgroups and namespaces and chroot of the container? We already have a short list of features that only work if the host node has some piece of software installed, this would be the same.

@ncdc

@ncdc

yes I've hit this several times. We added oc rsh that provides a helper for -itp -- bash but a more generic static linking solution would be good that works on any container

It can't totally depend on busybox (has to be replaceable/configurable) because we can't ship/support busybox.

Probably bonus points for allowing this to be more sophisticated down the road and inject other things (statically linked ssh)

@timstclair

Is this something we can prioritize?
The actual implementation, what would it look like?
A flag to the kubelet, sigh, that let's the user choose shell intepreter on host (default /bin/sh, since it exists nearly everywhere) and when the user is running kubectl sh it would just run nsenter, but with sh from the host, right?

cc @timstclair

Anyone up for this?

I don't think we'd want to override the image's shell, it would have to be a fallback iff the shell is missing.

@luxas

@luxas I have this problem as well, see #35584 for how I'd like to address it. I'm looking for additional use cases, so your input would be most welcome.

This isn't a priority for SIG Node (at least as defined in the tentative 2017Q1 road map), but it's the only thing I'm attempting to contribute so it's a priority for me.

I think I disagree with Clayton - if we offer this it should be consistent functionality. This doesn't have to replace exec, but it should work the same in all cases. Having a static busybox seems like a good start, but it does become one more thing that nodes need to have installed or that we ship with kubernetes.

…

On Wed, Jan 4, 2017 at 1:43 PM, Lee Verberne ***@***.***> wrote: @luxas <https://github.com/luxas> I have this problem as well, see #35584 <#35584> for how I'd like to address it. I'm looking for additional use cases, so your input would be most welcome. This isn't a priority for SIG Node (at least as defined in the tentative 2017Q1 road map), but it's the only thing I'm attempting to contribute so it's a priority for me. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10834 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVDAldH-SrWZegijl9qK_V_pa6NkRks5rPBKLgaJpZM4FTpIs> .

Let me clarify - it is unacceptable for exec to change to magically override the built in shell, because that is not backwards compatible and breaks large classes of apps on Kube. It's acceptable to make static injection a net new mechanism, and might be acceptable to fallback.

I also don't want a generic shell command that is tied too closely with one particular shell impl - we would not be able to ship that for lots of reasons, most of which include support. So I'm -1 to a "special" exec command that uses a shell binary we ship always.

I didn't mean to imply overriding. IMO `exec` should stay literal - run this command in the user's container. If that command doesn't exist, fail. This proposal was always about adding a net new shell API that was guaranteed to work on any container without the container needing to have any particular files in its image. So if you're -1 on that, can you explain why? The main reason many people do `FROM busybox` or `FROM alpine` is to have a shell and basic tools, but as soon as they do that they are beholden to every CVE in those not-my-app layers. It seems best to me to absolve users of the need to do that, if possible.

…

On Thu, Jan 5, 2017 at 8:00 AM, Clayton Coleman ***@***.***> wrote: Let me clarify - it is unacceptable for exec to change to magically override the built in shell, because that is not backwards compatible and breaks large classes of apps on Kube. It's acceptable to make static injection a net new mechanism, and *might* be acceptable to fallback. I also don't want a generic shell command that is tied too closely with one particular shell impl - we would not be able to ship that for lots of reasons, most of which include support. So I'm -1 to a "special" exec command that uses a shell binary we ship always. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10834 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVMQn3MvMbIRjl2ZNle7DO5qUOeNmks5rPROxgaJpZM4FTpIs> .

It's a requirement for the runtimes rather than the kubelet, and the CRI will need to add support. It's the run times that would have to ship a binary. Implementation would be pretty straightforward for the current container-based runtimes and slightly more difficult for hypervisor/cloud runtimes.

This solution is constrained to a single static binary specified by the Kubernetes developers, limiting its usefulness. I'd much prefer to be able to use an arbitrary container image for debugging, but the two aren't mutually exclusive.

I'm ok with a flag that says "inject shell" that makes it opt in. But we (red hat) probably won't support busybox for that, so my personal ask is that we try to define what API injected shell is expected to follow before we introduce it - has to be really carefully thought through. Agree with Lee's points that this won't work with all runtimes, and that the real problem isn't just a shell binary, it's the other tools (gdb, crash dump analysis, tar) that really become important. I'd probably lean towards a solution like what Lee is proposing than this.

…

On Thu, Jan 5, 2017 at 12:23 PM, Lee Verberne ***@***.***> wrote: It's a requirement for the runtimes rather than the kubelet, and the CRI will need to add support. It's the run times that would have to ship a binary. Implementation would be pretty straightforward for the current container-based runtimes and slightly more difficult for hypervisor/cloud runtimes. This solution is constrained to a single static binary specified by the Kubernetes developers, limiting its usefulness. I'd much prefer to be able to use an arbitrary container image for debugging, but the two aren't mutually exclusive. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10834 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p-0ixjpQVSQ9ekBmGTsBtaPgvhrvks5rPScGgaJpZM4FTpIs> .

I'm fine with more robust solutions - this was filed as an idea to solve the most immediate piece of the problem, but it's just an idea :) @smarterclayton should know by now that 96.3% of my ideas are outright terrible or at the very least are least off-target. On Thu, Jan 5, 2017 at 10:16 AM, Clayton Coleman <notifications@github.com> wrote:

…

I'm ok with a flag that says "inject shell" that makes it opt in. But we (red hat) probably won't support busybox for that, so my personal ask is that we try to define what API injected shell is expected to follow before we introduce it - has to be really carefully thought through. Agree with Lee's points that this won't work with all runtimes, and that the real problem isn't just a shell binary, it's the other tools (gdb, crash dump analysis, tar) that really become important. I'd probably lean towards a solution like what Lee is proposing than this. On Thu, Jan 5, 2017 at 12:23 PM, Lee Verberne ***@***.***> wrote: > It's a requirement for the runtimes rather than the kubelet, and the CRI > will need to add support. It's the run times that would have to ship a > binary. Implementation would be pretty straightforward for the current > container-based runtimes and slightly more difficult for hypervisor/cloud > runtimes. > > This solution is constrained to a single static binary specified by the > Kubernetes developers, limiting its usefulness. I'd much prefer to be able > to use an arbitrary container image for debugging, but the two aren't > mutually exclusive. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/kubernetes/issues/ 10834#issuecomment-270702682>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABG_p- 0ixjpQVSQ9ekBmGTsBtaPgvhrvks5rPScGgaJpZM4FTpIs> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10834 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVC28WNIKs6o7QomMBiRkFeg1aF95ks5rPTN_gaJpZM4FTpIs> .

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

/lifecycle frozen

With #59484 merged, this has become "possible". We still have a way to go to make it useful out of the box and to graduate the feature. I'm going to close this issue because we have a few others tracking this.

If you're interested in following along, keep an eye on issue #27140 and kubernetes/enhancements#277.

/close

@verb

@verb: Closing this issue.

Details

In response to this:

With #59484 merged, this has become "possible". We still have a way to go to make it useful out of the box and to graduate the feature. I'm going to close this issue because we have a few others tracking this.

If you're interested in following along, keep an eye on issue #27140 and kubernetes/enhancements#277.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

feat(config): convenience class with static constants for referencing…

Navigation Menu

Repository navigation

Make it possible to shell into any container, even if it has no shell #10834

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make it possible to shell into any container, even if it has no shell #10834

Description

Activity

smarterclayton commented on Aug 4, 2015

luxas commented on Jan 4, 2017

smarterclayton commented on Jan 4, 2017

verb commented on Jan 4, 2017

thockin commented on Jan 5, 2017

smarterclayton commented on Jan 5, 2017

thockin commented on Jan 5, 2017

verb commented on Jan 5, 2017

smarterclayton commented on Jan 5, 2017

thockin commented on Jan 5, 2017

38 remaining items

fejta-bot commented on Apr 15, 2019

zanetworker commented on May 12, 2019

fejta-bot commented on Aug 10, 2019

verb commented on Aug 13, 2019

tallclair commented on Aug 13, 2019

verb commented on Aug 27, 2019

k8s-ci-robot commented on Aug 27, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions