websocket error

[lly835]

[loganhz]

@lly835 the error code is 500. Please look at what the error is in chrome network tab and console log tab

[pjmazenot]

I'm getting this error too.

Here is what I get in the Network tab :
(Opcode -1)

And in the console log:
component.js:102 WebSocket connection to 'wss://[container_url]/exec?container=[container_name]&stdout=1&stdin=1&stderr=1&tty=1&command=%2Fbin%2Fsh&command=-c&command=TERM%3Dxterm-256color%3B%20export%20TERM%3B%20%5B%20-x%20%2Fbin%2Fbash%20%5D%20%26%26%20(%5B%20-x%20%2Fusr%2Fbin%2Fscript%20%5D%20%26%26%20%2Fusr%2Fbin%2Fscript%20-q%20-c%20%22%2Fbin%2Fbash%22%20%2Fdev%2Fnull%20%7C%7C%20exec%20%2Fbin%2Fbash)%20%7C%7C%20exec%20%2Fbin%2Fsh' failed: Error during WebSocket handshake: Unexpected response code: 500

It seems to be node related (vs container related) because for the same workloads on different nodes if I request the shell it works for some nodes and some not.

EDIT: The calico-node_canal container is caught in a restart loop on the nodes that are not working

Installation info

Rancher versions:
rancher/rancher: v2.0.8
rancher/rancher-agent: v2.0.8

Infrastructure Stack versions:
healthcheck:
ipsec:
network-services:
scheduler:
kubernetes (if applicable): v1.10.1

Docker version: (docker version,docker info preferred)
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 03:35:14 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 03:35:14 2017
OS/Arch: linux/amd64
Experimental: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
4.4.127-mainline-rev1

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Bare-metal

Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
HA rancher, internal DB

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Cattle

[leodotcloud]

Possibly:

• Internal/External ipaddress missing for hosts in rhel clusters. #14600
• One Node loose private and external ip address kubernetes/kubernetes#68270

Run kubectl get nodes -o wide to see if the node, where the pod is running, is missing either internal or external or both ip addresses.

[bradjones1]

@leodotcloud I'm running into a similar issue. Imported GKE clusters, and requests to tail logs or exec into containers results in a 500 error. The cluster is otherwise operational and I don't see anything particularly telling in the Rancher logs.

[bradjones1]

Same as #13149

[bradjones1]

On further review I do notice, however, that the clusters which are not able to be exec'd to have similar reports in the Rancher logs (repeated multiple times for different resource types):

*v1.NetworkPolicy: Get https://10.0.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect

The IP matches the kubernetes service cluster IP.

[bradjones1]

Both of the affected clusters previously had calico network policy support enabled, prior to it being disabled for non-RKE clusters.

[bradjones1]

Fixed in my case: The issue was with a missing firewall rule for the VPC on GCP to allow ssh connections from the master to nodes. It would be great to see some more helpful logging from Rancher on the failure's origin, but this is not a Rancher error, at least in my case. https://stackoverflow.com/a/36378625

[Allen-yan]

I have the same issue , when I ran pod on a special node .
My 2 rancher instances both have the issue.
Rancher:2.0.8 / 2.1.5

The node had a vpn client . And I can see 2 IP address on the rancher UI

bh-k8s-worker-test03
10.10.20.53  /  192.168.42.10

All the pods run on this node can not access to Shell or Log

On the node I input ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.20.53  netmask 255.255.255.0  broadcast 10.10.20.255
        ether 00:16:3e:02:fc:7a  txqueuelen 1000  (Ethernet)
        RX packets 35991105  bytes 23970906975 (22.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25027690  bytes 18028073674 (16.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1193506  bytes 558015315 (532.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1193506  bytes 558015315 (532.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1280
        inet 192.168.42.10  netmask 255.255.255.255  destination 192.168.42.1
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 6902  bytes 2912463 (2.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6591  bytes 467362 (456.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunl0: flags=193<UP,RUNNING,NOARP>  mtu 1440
        inet 10.42.1.1  netmask 255.255.255.255
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 5197120  bytes 18320006318 (17.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4288239  bytes 14157927220 (13.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[VickyKung]

have you any way to solve it ?