can not execute command in a container using kubectl exec

[divyenpatel]

Deployed K8S 1.5.3 cluster using kubernetes-anywhere latest code

$ kubectl cluster-info
Kubernetes master is running at https://10.192.36.202
Heapster is running at https://10.192.36.202/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://10.192.36.202/api/v1/proxy/namespaces/kube-system/services/kube-dns

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:40:50Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} 

Created pod using following YAML

$ cat injectpod.yaml 
apiVersion: v1
kind: Pod
metadata:
    name: inject-pod
spec:
    containers:
    - name: test-container
      image: gcr.io/google_containers/busybox:1.24
      command: ["/bin/sh", "-c", "echo 'hello' > /mnt/volume1/index.html  && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done"]
      volumeMounts:
      - name: test-volume
        mountPath: /mnt/volume1
    securityContext:
      seLinuxOptions:
        level: "s0:c0,c1"
    restartPolicy: Never
    volumes:
    - name: test-volume
      vsphereVolume:
          volumePath: "[vsanDatastore] kubevols/my-vmdk.vmdk"
          fsType: ext4
$ 

$ kubectl create -f injectpod.yaml
pod "inject-pod" created
$

$ kubectl get pods
NAME         READY     STATUS    RESTARTS   AGE
inject-pod   1/1       Running   0          22s

$ kubectl describe pods inject-pod
Name:		inject-pod
Namespace:	default
Node:		node1/10.192.40.58
Start Time:	Fri, 17 Feb 2017 17:08:50 -0800
Labels:		<none>
Status:		Running
IP:		172.1.53.3
Controllers:	<none>
Containers:
  test-container:
    Container ID:	docker://b0616f1347dc5b02b4fdb81d52b7c75be3a77393ab5f0b1f9c6a75ec85f19232
    Image:		gcr.io/google_containers/busybox:1.24
    Image ID:		docker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9
    Port:		
    Command:
      /bin/sh
      -c
      echo 'hello' > /mnt/volume1/index.html  && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done
    State:		Running
      Started:		Fri, 17 Feb 2017 17:09:02 -0800
    Ready:		True
    Restart Count:	0
    Volume Mounts:
      /mnt/volume1 from test-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mcqgg (ro)
    Environment Variables:	<none>
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	True 
  PodScheduled 	True 
Volumes:
  test-volume:
    Type:	vSphereVolume (a Persistent Disk resource in vSphere)
    VolumePath:	[vsanDatastore] kubevols/my-vmdk.vmdk
    FSType:	ext4
  default-token-mcqgg:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-mcqgg
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath			Type		Reason		Message
  ---------	--------	-----	----			-------------			--------	------		-------
  5m		5m		1	{default-scheduler }					Normal		Scheduled	Successfully assigned inject-pod to node1
  5m		5m		1	{kubelet node1}		spec.containers{test-container}	Normal		Pulling		pulling image "gcr.io/google_containers/busybox:1.24"
  5m		5m		1	{kubelet node1}		spec.containers{test-container}	Normal		Pulled		Successfully pulled image "gcr.io/google_containers/busybox:1.24"
  5m		5m		1	{kubelet node1}		spec.containers{test-container}	Normal		Created		Created container with docker id b0616f1347dc; Security:[seccomp=unconfined]
  5m		5m		1	{kubelet node1}		spec.containers{test-container}	Normal		Started		Started container with docker id b0616f1347dc
$

When tried to execute command on the container of inject-pod, kubectl exec failed with error: unable to upgrade connection: pod does not exist

$ kubectl exec -ti inject-pod -- bin/sh
error: unable to upgrade connection: pod does not exist

$ kubectl --server=https://10.192.36.202 --kubeconfig=/Users/divyenp/kanywhere1.5.3/kubernetes-anywhere/phase1/vsphere/.tmp/kubeconfig.json exec inject-pod -- /bin/ls -ld /mnt/volume1
error: unable to upgrade connection: pod does not exist

When tried to execute command using docker exec it worked.

root@photon-zv1KbtvMG [ ~ ]# docker exec -it b0616f1347dc bin/sh
/ # ls -ld /mnt/volume1
drwxr-xr-x    3 root     root          4096 Feb 18 01:09 /mnt/volume1
/ # 
/ # 
/ # 
/ # cat /mnt/volume1/index.html 
hello
/ # 

verified --enable-debugging-handlers is set for kubelet in master and all workder node.
kubectl config from worker node

# ps auxw | grep kubelet
root       616  0.0  0.6 143828 27880 ?        Ssl  00:44   0:00 /usr/bin/docker run --net=host --pid=host --privileged -v /dev:/dev -v /sys:/sys:ro -v /var/run:/var/run:rw -v /var/lib/docker/:/var/lib/docker:rw -v /var/lib/kubelet/:/var/lib/kubelet:shared -v /var/log:/var/log:shared -v /srv/kubernetes:/srv/kubernetes:ro -v /etc/kubernetes:/etc/kubernetes:ro gcr.io/google-containers/hyperkube-amd64:v1.5.3 /hyperkube kubelet --address=0.0.0.0 --allow-privileged=true --cloud-provider=vsphere --enable-server --enable-debugging-handlers --kubeconfig=/srv/kubernetes/kubeconfig.json --config=/etc/kubernetes/manifests --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --v=2 --api-servers=https://10.192.36.202 --hairpin-mode=promiscuous-bridge --cloud-config=/etc/kubernetes/vsphere.conf
root       738  1.6  2.2 372656 92260 ?        Ssl  00:48   0:42 /hyperkube kubelet --address=0.0.0.0 --allow-privileged=true --cloud-provider=vsphere --enable-server --enable-debugging-handlers --kubeconfig=/srv/kubernetes/kubeconfig.json --config=/etc/kubernetes/manifests --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --v=2 --api-servers=https://10.192.36.202 --hairpin-mode=promiscuous-bridge --cloud-config=/etc/kubernetes/vsphere.conf
root     28476  0.0  0.0   6484   876 pts/0    S+   01:30   0:00 grep --color=auto kubelet

kubectl config from master node

# ps auxw | grep kubelet
root       656  0.0  0.6  87544 27760 ?        Ssl  00:45   0:00 /usr/bin/docker run --net=host --pid=host --privileged -v /dev:/dev -v /sys:/sys:ro -v /var/run:/var/run:rw -v /var/lib/docker/:/var/lib/docker:rw -v /var/lib/kubelet/:/var/lib/kubelet:shared -v /var/log:/var/log:shared -v /srv/kubernetes:/srv/kubernetes:ro -v /etc/kubernetes:/etc/kubernetes:ro gcr.io/google-containers/hyperkube-amd64:v1.5.3 /hyperkube kubelet --address=0.0.0.0 --allow-privileged=true --cloud-provider=vsphere --enable-server --enable-debugging-handlers --kubeconfig=/srv/kubernetes/kubeconfig.json --config=/etc/kubernetes/manifests --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --v=2 --api-servers=http://localhost:8080 --register-schedulable=false --cloud-config=/etc/kubernetes/vsphere.conf
root       770  2.0  2.2 511924 91396 ?        Ssl  00:48   0:53 /hyperkube kubelet --address=0.0.0.0 --allow-privileged=true --cloud-provider=vsphere --enable-server --enable-debugging-handlers --kubeconfig=/srv/kubernetes/kubeconfig.json --config=/etc/kubernetes/manifests --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --v=2 --api-servers=http://localhost:8080 --register-schedulable=false --cloud-config=/etc/kubernetes/vsphere.conf
root     29751  0.0  0.0   6484   888 pts/0    S+   01:31   0:00 grep --color=auto kubelet

Expectation
After deployment user should be able to execute command in the pod container using kubectl exec

cc: @kerneltime @abrarshivani @BaluDontu @tusharnt @pdhamdhere

[divyenpatel]

K8S-1.5.3-Deployment-Logs.txt

[kerneltime]

Looks like a networking setup issue. I don't know the exact cause for this but the apiserver and proxy are trying to resolve the hostname which is the same on all nodes and not really stored with etcd.

2017-02-15T06:08:31.937282395Z E0215 06:08:31.936655       1 server.go:421] Can't get Node "photon-zv1kbtvmg", assuming iptables proxy, err: nodes "photon-zv1kbtvmg" not found
2017-02-15T06:08:32.377187394Z I0215 06:08:31.939499       1 server.go:215] Using iptables Proxier.
2017-02-15T06:08:32.377192031Z W0215 06:08:31.943176       1 server.go:468] Failed to retrieve node info: nodes "photon-zv1kbtvmg" not found
2017-02-15T06:08:32.377194769Z W0215 06:08:31.943320       1 proxier.go:249] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
2017-02-15T06:08:32.377197751Z W0215 06:08:31.943330       1 proxier.go:254] clusterCIDR not specified, unable to distinguish between internal and external traffic
2017-02-15T06:08:32.377200405Z I0215 06:08:31.943401       1 server.go:227] Tearing down userspace rules.
2017-02-15T06:08:32.377202910Z I0215 06:08:31.986846       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
2017-02-15T06:08:32.377205631Z I0215 06:08:31.990823       1 conntrack.go:66] Setting conntrack hashsize to 32768
2017-02-15T06:08:32.377208342Z I0215 06:08:31.991342       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2017-02-15T06:08:32.377210900Z I0215 06:08:31.991372       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

I went ahead and changed the hostname on all nodes to match the node names and exec is now working.
We have an existing task to set the hostname to be the node name, once we close that this issue should be resolved but we need to understand the change between 1.4 and 1.5 that leads to this behavior.

./kubectl.1.5 exec redis-master-3729132442-pjsp7 date
Sat Feb 18 02:27:51 UTC 2017

[kerneltime]

related #295

[colemickens]

(Just skimming but this might be related: kubernetes/kubernetes#33718)

[kerneltime]

I guess then we need to figure out why the hostname gets picked up (which we do not customize) instead of the vm node name returned.