Ephemeral containers using kubectl-debug does not seem to work

[howardjohn]

What happened:

Using https://github.com/aylei/kubectl-debug, ephemeral contains do not run properly on Kind. The container is created but then it hangs - not sure where exactly its going wrong.

What you expected to happen:

Shell opens up in ephemeral container. This occurs on other Kubernetes (at least GKE 1.16)

How to reproduce it (as minimally and precisely as possible):

kind create cluster
kubectl apply -f SOME_POD.yaml
kubectl-debug pod_name

Anything else we need to know?:

It could be an issue with kubectl-debug -- its a prototype of what the command may look like so it could have some bugs.

Environment:

• kind version: (use kind version): kind v0.6.0 go1.13 linux/amd64
• Kubernetes version: (use kubectl version): 1.16.3
• Docker version: (use docker info): 19.03.5
• OS (e.g. from /etc/os-release): linux

[BenTheElder]

/assign

[BenTheElder]

This is a problem with the tool.

aylei/kubectl-debug#3

If you inspect the debug agent pod, you'll find that it mounts /var/run/docker.sock.
This tool speaks to dockerd, it doesn't seem to use CRI (which also suggests that it isn't using kubernetes ephemeral containers)

[howardjohn]

Huh, I seem to recall that being linked to as an example of kubectl support for ephemeral containers, but you are right looks like it is not. Thanks for the follow up. Sounds like no action needed here, so feel free to close unless you prefer to keep open for future reference

[BenTheElder]

Actual ephemeral containers are behind a feature gate in 1.16+ and won't not be until they are out of alpha.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/#examples

Huh, I seem to recall that being linked to as an example of kubectl support for ephemeral containers, but you are right looks like it is not. Thanks for the follow up.

There is a KEP for kubectl debug in-tree but it doesn't look related to that tool and the KEP is not merged yet. kubernetes/enhancements#1204

Sounds like no action needed here, so feel free to close unless you prefer to keep open for future reference

I want to go ahead and confirm if ephemeral containers work first while we're at it, will need to create a 1.16+ cluster with the featuregate and create one.

[howardjohn]

At some point I had a ephemeral container created (in kind), but I couldn't actually attach to it. But the pod spec showed it running. I thought I was using that tool, not sure if it change since then (it was right when 1.16 came out that I tried this). This may have been user error though, not sure.

[BenTheElder]

It seems to work with the latest kind:

1. create kind cluster with feature gate enabled

cat <<EOF | kind create cluster --config=- 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
# enable EphemeralContainers feature gate
- |
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  scheduler:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  controllerManager:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: InitConfiguration
  metadata:
    name: config
  nodeRegistration:
    kubeletExtraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  kind: KubeletConfiguration
  featureGates:
    EphemeralContainers: true
- |
  kind: KubeProxyConfiguration
  featureGates:
    EphemeralContainers: true
EOF

2. Create example pod

cat <<EOF | kubectl create -f - 
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - image: mhausenblas/simpleservice:0.5.0
    name: app
EOF

3. Update ephemeral containers:

cat <<EOF | kubectl replace --raw /api/v1/namespaces/default/pods/example-pod/ephemeralcontainers -f -
{
    "apiVersion": "v1",
    "kind": "EphemeralContainers",
    "metadata": {
            "name": "example-pod"
    },
    "ephemeralContainers": [{
        "command": [
            "sh"
        ],
        "image": "busybox",
        "imagePullPolicy": "IfNotPresent",
        "name": "debugger",
        "stdin": true,
        "tty": true,
        "terminationMessagePolicy": "File"
    }]
}
EOF

4. Attach
   kubectl attach -it example-pod -c debugger

It's certainly clunky this way though, and without PID namespace sharing on it's not quite as helpful (though this could also be turned on).

[BenTheElder]

in the next release this will just be:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 EphemeralContainers: true

(except for the pid namespace sharing bit...)

[ledroide]

in the next release this will just be:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 EphemeralContainers: true

Hello @BenTheElder . Would you know some documentation about this way of configuring feature-gates, and starting since which kubernetes version ? I can't find it