Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ephemeral containers using kubectl-debug does not seem to work #1210

Closed
howardjohn opened this issue Dec 27, 2019 · 8 comments
Closed

Ephemeral containers using kubectl-debug does not seem to work #1210

howardjohn opened this issue Dec 27, 2019 · 8 comments
Assignees
Labels
kind/external upstream bugs

Comments

@howardjohn
Copy link
Contributor

howardjohn commented Dec 27, 2019

What happened:

Using https://github.com/aylei/kubectl-debug, ephemeral contains do not run properly on Kind. The container is created but then it hangs - not sure where exactly its going wrong.

What you expected to happen:

Shell opens up in ephemeral container. This occurs on other Kubernetes (at least GKE 1.16)

How to reproduce it (as minimally and precisely as possible):

kind create cluster
kubectl apply -f SOME_POD.yaml
kubectl-debug pod_name

Anything else we need to know?:

It could be an issue with kubectl-debug -- its a prototype of what the command may look like so it could have some bugs.

Environment:

  • kind version: (use kind version): kind v0.6.0 go1.13 linux/amd64
  • Kubernetes version: (use kubectl version): 1.16.3
  • Docker version: (use docker info): 19.03.5
  • OS (e.g. from /etc/os-release): linux
@howardjohn howardjohn added the kind/bug Categorizes issue or PR as related to a bug. label Dec 27, 2019
@BenTheElder
Copy link
Member

BenTheElder commented Jan 2, 2020

/assign

@BenTheElder
Copy link
Member

BenTheElder commented Jan 2, 2020

This is a problem with the tool.

aylei/kubectl-debug#3

If you inspect the debug agent pod, you'll find that it mounts /var/run/docker.sock.
This tool speaks to dockerd, it doesn't seem to use CRI (which also suggests that it isn't using kubernetes ephemeral containers)

@BenTheElder BenTheElder added kind/external upstream bugs and removed kind/bug Categorizes issue or PR as related to a bug. labels Jan 2, 2020
@howardjohn
Copy link
Contributor Author

howardjohn commented Jan 2, 2020

Huh, I seem to recall that being linked to as an example of kubectl support for ephemeral containers, but you are right looks like it is not. Thanks for the follow up. Sounds like no action needed here, so feel free to close unless you prefer to keep open for future reference

@BenTheElder
Copy link
Member

BenTheElder commented Jan 2, 2020

Actual ephemeral containers are behind a feature gate in 1.16+ and won't not be until they are out of alpha.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/#examples

Huh, I seem to recall that being linked to as an example of kubectl support for ephemeral containers, but you are right looks like it is not. Thanks for the follow up.

There is a KEP for kubectl debug in-tree but it doesn't look related to that tool and the KEP is not merged yet. kubernetes/enhancements#1204

Sounds like no action needed here, so feel free to close unless you prefer to keep open for future reference

I want to go ahead and confirm if ephemeral containers work first while we're at it, will need to create a 1.16+ cluster with the featuregate and create one.

@howardjohn
Copy link
Contributor Author

howardjohn commented Jan 2, 2020

At some point I had a ephemeral container created (in kind), but I couldn't actually attach to it. But the pod spec showed it running. I thought I was using that tool, not sure if it change since then (it was right when 1.16 came out that I tried this). This may have been user error though, not sure.

@BenTheElder
Copy link
Member

BenTheElder commented Jan 2, 2020

It seems to work with the latest kind:

  1. create kind cluster with feature gate enabled
cat <<EOF | kind create cluster --config=- 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
# enable EphemeralContainers feature gate
- |
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  scheduler:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  controllerManager:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: InitConfiguration
  metadata:
    name: config
  nodeRegistration:
    kubeletExtraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: KubeletConfiguration
  featureGates:
    EphemeralContainers: true
- |
  kind: KubeProxyConfiguration
  featureGates:
    EphemeralContainers: true
EOF
  1. Create example pod
cat <<EOF | kubectl create -f - 
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - image: mhausenblas/simpleservice:0.5.0
    name: app
EOF
  1. Update ephemeral containers:
cat <<EOF | kubectl replace --raw /api/v1/namespaces/default/pods/example-pod/ephemeralcontainers -f -
{
    "apiVersion": "v1",
    "kind": "EphemeralContainers",
    "metadata": {
            "name": "example-pod"
    },
    "ephemeralContainers": [{
        "command": [
            "sh"
        ],
        "image": "busybox",
        "imagePullPolicy": "IfNotPresent",
        "name": "debugger",
        "stdin": true,
        "tty": true,
        "terminationMessagePolicy": "File"
    }]
}
EOF
  1. Attach
    kubectl attach -it example-pod -c debugger

It's certainly clunky this way though, and without PID namespace sharing on it's not quite as helpful (though this could also be turned on).

@BenTheElder
Copy link
Member

BenTheElder commented Apr 16, 2020

in the next release this will just be:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 EphemeralContainers: true

(except for the pid namespace sharing bit...)

@ledroide
Copy link

ledroide commented Feb 4, 2021

in the next release this will just be:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 EphemeralContainers: true

Hello @BenTheElder . Would you know some documentation about this way of configuring feature-gates, and starting since which kubernetes version ? I can't find it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/external upstream bugs
Projects
None yet
Development

No branches or pull requests

3 participants