Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

default backend-404, follow the official tutorial. #26385

Closed
3AceShowHand opened this issue Mar 31, 2020 · 10 comments
Closed

default backend-404, follow the official tutorial. #26385

3AceShowHand opened this issue Mar 31, 2020 · 10 comments
Labels
status/more-info

Comments

@3AceShowHand
Copy link

3AceShowHand commented Mar 31, 2020 •
edited
Loading

There are 4 issues like this one, but none of them solved my problem. I cannot access the UI after create a rancher server cluster, default backend - 404 returned.

A: Nginx L4 load balancer

worker_processes 4;
worker_rlimit_nofile 40000;

events {
  worker_connections 8192;
}

stream {
  upstream rancher_servers_http {
    least_conn;
    server 172.16.5.187:80 max_fails=3 fail_timeout=5s;
    server 172.16.5.116:80 max_fails=3 fail_timeout=5s;
    server 172.16.5.118:80 max_fails=3 fail_timeout=5s;
  }
  server {
    listen     80;
    proxy_pass rancher_servers_http;
  }

  upstream rancher_servers_https {
    least_conn;
    server 172.16.5.187:443 max_fails=3 fail_timeout=5s;
    server 172.16.5.116:443 max_fails=3 fail_timeout=5s;
    server 172.16.5.118:443 max_fails=3 fail_timeout=5s;
  }
  server {
    listen     443;
    proxy_pass rancher_servers_https;
  }
}

B、C、D：rancher server cluster.

sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

systemctl disable firewalld
yum remove firewalld -y
yum install -y iptables-services
systemctl enable iptables
iptables -F
sevice iptables save
systemctl restart iptables

cat >> /etc/sysctl.conf<<EOF
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
EOF

all are vms provided by IT engineer, centos 7.6.
What kind of request is this (question/bug/enhancement/feature request):
question / bug / help
Steps to reproduce (least amount of steps as possible):
`

  • rke up --config rancher-cluster.yml
  • export KUBECONFIG=./kube_config_rancher-cluster.yml
  • helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
  • kubectl create namespace cattle-system
  • helm repo update
  • helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.my.org --set ingress.tls.source=secret
  • kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key
  • kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem
  • kubectl -n cattle-system rollout status deploy/rancher
    `

Result:
cannot open UI by enter the hostname, return default backend -404

Other details that may be helpful:
getenforce: Disabled
helm version: v3.1.2
rke version: v1.1.0
kubectl version: 1.17

why only 2 canel-xxx ?

 kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   rancher-6bc58865f9-6b598                  1/1     Running     0          14m
cattle-system   rancher-6bc58865f9-c97v2                  1/1     Running     0          14m
cattle-system   rancher-6bc58865f9-k6hgz                  1/1     Running     0          14m
ingress-nginx   default-http-backend-67cf578fc4-6hmpg     1/1     Running     0          17m
ingress-nginx   nginx-ingress-controller-72c94            1/1     Running     0          17m
ingress-nginx   nginx-ingress-controller-999nz            1/1     Running     0          17m
ingress-nginx   nginx-ingress-controller-prnl7            1/1     Running     0          17m
kube-system     canal-hql4t                               2/2     Running     0          17m
kube-system     canal-kkb5z                               2/2     Running     0          17m
kube-system     canal-tx8k2                               2/2     Running     0          17m
kube-system     coredns-7c5566588d-5zrlk                  1/1     Running     0          17m
kube-system     coredns-7c5566588d-sfrgx                  1/1     Running     0          17m
kube-system     coredns-autoscaler-65bfc8d47d-bxs6d       1/1     Running     0          17m
kube-system     metrics-server-6b55c64f86-xk45f           1/1     Running     0          17m
kube-system     rke-coredns-addon-deploy-job-rfn22        0/1     Completed   0          17m
kube-system     rke-ingress-controller-deploy-job-z78kw   0/1     Completed   0          17m
kube-system     rke-metrics-addon-deploy-job-z8d8x        0/1     Completed   0          17m
kube-system     rke-network-plugin-deploy-job-t594g       0/1     Completed   0          17m

all output by following commands on https://rancher.com/docs/rancher/v2.x/en/troubleshooting/rancherha/

kubectl -n cattle-system get pods -l app=rancher -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP          NODE           NOMINATED NODE   READINESS GATES
rancher-6bc58865f9-6b598   1/1     Running   0          15m   10.42.1.7   172.16.5.118   <none>           <none>
rancher-6bc58865f9-c97v2   1/1     Running   0          15m   10.42.0.5   172.16.5.116   <none>           <none>
rancher-6bc58865f9-k6hgz   1/1     Running   0          15m   10.42.2.4   172.16.5.187   <none>           <none>

This time, rancher didn't restart 1 or 2 times, this is wired.

kubectl -n cattle-system describe pods -l app=rancher
Name:         rancher-6bc58865f9-6b598
Namespace:    cattle-system
Priority:     0
Node:         172.16.5.118/172.16.5.118
Start Time:   Tue, 31 Mar 2020 21:54:26 +0800
Labels:       app=rancher
              pod-template-hash=6bc58865f9
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.1.7/32
              cni.projectcalico.org/podIPs: 10.42.1.7/32
Status:       Running
IP:           10.42.1.7
IPs:
  IP:           10.42.1.7
Controlled By:  ReplicaSet/rancher-6bc58865f9
Containers:
  rancher:
    Container ID:  docker://7ea84ea79b571cf4888007b323ab5039bf32cb7b61865e742c2fb136ce507fce
    Image:         rancher/rancher:v2.3.6
    Image ID:      docker-pullable://rancher/rancher@sha256:e6e7a9b87182b2298147118a756d0eab532768c0e92cb53a9a5a72a052b6cca7
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --http-listen-port=80
      --https-listen-port=443
      --add-local=auto
    State:          Running
      Started:      Tue, 31 Mar 2020 22:00:42 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /etc/rancher/ssl/cacerts.pem from tls-ca-volume (ro,path="cacerts.pem")
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-dk5g7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tls-ca-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tls-ca
    Optional:    false
  rancher-token-dk5g7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-dk5g7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                   Message
  ----     ------       ----               ----                   -------
  Normal   Scheduled    <unknown>          default-scheduler      Successfully assigned cattle-system/rancher-6bc58865f9-6b598 to 172.16.5.118
  Warning  FailedMount  15m (x8 over 16m)  kubelet, 172.16.5.118  MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
  Warning  FailedMount  14m                kubelet, 172.16.5.118  Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[rancher-token-dk5g7 tls-ca-volume]: timed out waiting for the condition
  Normal   Pulling      13m                kubelet, 172.16.5.118  Pulling image "rancher/rancher:v2.3.6"
  Normal   Pulled       9m55s              kubelet, 172.16.5.118  Successfully pulled image "rancher/rancher:v2.3.6"
  Normal   Created      9m55s              kubelet, 172.16.5.118  Created container rancher
  Normal   Started      9m55s              kubelet, 172.16.5.118  Started container rancher
  Warning  Unhealthy    9m50s              kubelet, 172.16.5.118  Readiness probe failed: Get http://10.42.1.7:80/healthz: dial tcp 10.42.1.7:80: connect: connection refused


Name:         rancher-6bc58865f9-c97v2
Namespace:    cattle-system
Priority:     0
Node:         172.16.5.116/172.16.5.116
Start Time:   Tue, 31 Mar 2020 21:54:26 +0800
Labels:       app=rancher
              pod-template-hash=6bc58865f9
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.0.5/32
              cni.projectcalico.org/podIPs: 10.42.0.5/32
Status:       Running
IP:           10.42.0.5
IPs:
  IP:           10.42.0.5
Controlled By:  ReplicaSet/rancher-6bc58865f9
Containers:
  rancher:
    Container ID:  docker://72fb2f75fcf93333b32424c8a2f7ef7d914c0ce36491ef9500bd26413c108bcc
    Image:         rancher/rancher:v2.3.6
    Image ID:      docker-pullable://rancher/rancher@sha256:e6e7a9b87182b2298147118a756d0eab532768c0e92cb53a9a5a72a052b6cca7
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --http-listen-port=80
      --https-listen-port=443
      --add-local=auto
    State:          Running
      Started:      Tue, 31 Mar 2020 22:00:28 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /etc/rancher/ssl/cacerts.pem from tls-ca-volume (ro,path="cacerts.pem")
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-dk5g7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tls-ca-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tls-ca
    Optional:    false
  rancher-token-dk5g7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-dk5g7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                   Message
  ----     ------       ----               ----                   -------
  Normal   Scheduled    <unknown>          default-scheduler      Successfully assigned cattle-system/rancher-6bc58865f9-c97v2 to 172.16.5.116
  Warning  FailedMount  15m (x8 over 16m)  kubelet, 172.16.5.116  MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
  Warning  FailedMount  14m                kubelet, 172.16.5.116  Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[tls-ca-volume rancher-token-dk5g7]: timed out waiting for the condition
  Normal   Pulling      13m                kubelet, 172.16.5.116  Pulling image "rancher/rancher:v2.3.6"
  Normal   Pulled       10m                kubelet, 172.16.5.116  Successfully pulled image "rancher/rancher:v2.3.6"
  Normal   Created      10m                kubelet, 172.16.5.116  Created container rancher
  Normal   Started      10m                kubelet, 172.16.5.116  Started container rancher
  Warning  Unhealthy    9m52s              kubelet, 172.16.5.116  Readiness probe failed: Get http://10.42.0.5:80/healthz: dial tcp 10.42.0.5:80: connect: connection refused


Name:         rancher-6bc58865f9-k6hgz
Namespace:    cattle-system
Priority:     0
Node:         172.16.5.187/172.16.5.187
Start Time:   Tue, 31 Mar 2020 21:54:26 +0800
Labels:       app=rancher
              pod-template-hash=6bc58865f9
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.2.4/32
              cni.projectcalico.org/podIPs: 10.42.2.4/32
Status:       Running
IP:           10.42.2.4
IPs:
  IP:           10.42.2.4
Controlled By:  ReplicaSet/rancher-6bc58865f9
Containers:
  rancher:
    Container ID:  docker://98693f13c5095c3441f2c415fb17d4c1421bc3508404384dde39a899574aede8
    Image:         rancher/rancher:v2.3.6
    Image ID:      docker-pullable://rancher/rancher@sha256:e6e7a9b87182b2298147118a756d0eab532768c0e92cb53a9a5a72a052b6cca7
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --http-listen-port=80
      --https-listen-port=443
      --add-local=auto
    State:          Running
      Started:      Tue, 31 Mar 2020 22:00:59 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /etc/rancher/ssl/cacerts.pem from tls-ca-volume (ro,path="cacerts.pem")
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-dk5g7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tls-ca-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tls-ca
    Optional:    false
  rancher-token-dk5g7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-dk5g7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                   Message
  ----     ------       ----               ----                   -------
  Normal   Scheduled    <unknown>          default-scheduler      Successfully assigned cattle-system/rancher-6bc58865f9-k6hgz to 172.16.5.187
  Warning  FailedMount  15m (x8 over 16m)  kubelet, 172.16.5.187  MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
  Warning  FailedMount  14m                kubelet, 172.16.5.187  Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[rancher-token-dk5g7 tls-ca-volume]: timed out waiting for the condition
  Normal   Pulling      13m                kubelet, 172.16.5.187  Pulling image "rancher/rancher:v2.3.6"
  Normal   Pulled       9m39s              kubelet, 172.16.5.187  Successfully pulled image "rancher/rancher:v2.3.6"
  Normal   Created      9m39s              kubelet, 172.16.5.187  Created container rancher
  Normal   Started      9m38s              kubelet, 172.16.5.187  Started container rancher

kubectl -n cattle-system logs -l app=rancher
2020/03/31 14:01:19 [INFO] Deleting roleBinding clusterrolebinding-gq58m
2020/03/31 14:01:19 [INFO] Deleting roleBinding clusterrolebinding-jxhbt
2020/03/31 14:01:19 [INFO] Creating roleBinding User u-o7q3cttxk3 Role edit
2020/03/31 14:01:19 [INFO] Creating roleBinding User u-o7q3cttxk3 Role project-member
2020/03/31 14:01:19 [ERROR] ProjectController local/p-47srp [system-image-upgrade-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:21 [ERROR] ProjectController local/p-47srp [system-image-upgrade-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:21 [ERROR] ProjectController local/p-dxvxh [system-image-upgrade-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:21 [ERROR] CatalogController system-library [system-image-upgrade-catalog-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:25 [ERROR] CatalogController system-library [system-image-upgrade-catalog-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:29 [ERROR] ProjectController local/p-dxvxh [system-image-upgrade-controller] failed with : upgrade cluster local system service logging failed: cluster local not ready
2020/03/31 14:01:19 [INFO] Redeploy Rancher Agents is needed for local: forceDeploy=false, agent/auth image changed=true, private repo changed=false
2020/03/31 14:01:19 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace
2020/03/31 14:01:19 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace
2020/03/31 14:01:19 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2020/03/31 14:01:19 [INFO] Redeploy Rancher Agents is needed for local: forceDeploy=false, agent/auth image changed=true, private repo changed=false
2020/03/31 14:01:19 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2020/03/31 14:01:20 [INFO] Handling backend connection request [10.42.1.7]
2020/03/31 14:01:32 [INFO] Redeploy Rancher Agents is needed for local: forceDeploy=false, agent/auth image changed=true, private repo changed=false
2020/03/31 14:01:32 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2020/03/31 14:02:14 [INFO] Catalog-cache removed 2 entries from disk
2020/03/31 14:00:59 [INFO] Running in clustered mode with ID 10.42.2.4, monitoring endpoint cattle-system/rancher
2020/03/31 14:01:00 [INFO] Starting API controllers
2020/03/31 14:01:00 [INFO] Listening on :443
2020/03/31 14:01:00 [INFO] Listening on :80
2020/03/31 14:01:00 [INFO] Starting cluster controllers for local
2020/03/31 14:01:00 [INFO] Starting cluster agent for local [owner=false]
2020/03/31 14:01:15 [INFO] Adding peer wss://10.42.0.5/v3/connect, 10.42.0.5
2020/03/31 14:01:17 [INFO] Adding peer wss://10.42.1.7/v3/connect, 10.42.1.7
2020/03/31 14:01:17 [INFO] Handling backend connection request [10.42.1.7]
2020/03/31 14:01:17 [INFO] Handling backend connection request [10.42.0.5]

kubectl -n cattle-system get events
LAST SEEN   TYPE      REASON              OBJECT                          MESSAGE
<unknown>   Normal    Scheduled           pod/rancher-6bc58865f9-6b598    Successfully assigned cattle-system/rancher-6bc58865f9-6b598 to 172.16.5.118
19m         Warning   FailedMount         pod/rancher-6bc58865f9-6b598    MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
18m         Warning   FailedMount         pod/rancher-6bc58865f9-6b598    Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[rancher-token-dk5g7 tls-ca-volume]: timed out waiting for the condition
17m         Normal    Pulling             pod/rancher-6bc58865f9-6b598    Pulling image "rancher/rancher:v2.3.6"
13m         Normal    Pulled              pod/rancher-6bc58865f9-6b598    Successfully pulled image "rancher/rancher:v2.3.6"
13m         Normal    Created             pod/rancher-6bc58865f9-6b598    Created container rancher
13m         Normal    Started             pod/rancher-6bc58865f9-6b598    Started container rancher
13m         Warning   Unhealthy           pod/rancher-6bc58865f9-6b598    Readiness probe failed: Get http://10.42.1.7:80/healthz: dial tcp 10.42.1.7:80: connect: connection refused
<unknown>   Normal    Scheduled           pod/rancher-6bc58865f9-c97v2    Successfully assigned cattle-system/rancher-6bc58865f9-c97v2 to 172.16.5.116
19m         Warning   FailedMount         pod/rancher-6bc58865f9-c97v2    MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
18m         Warning   FailedMount         pod/rancher-6bc58865f9-c97v2    Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[tls-ca-volume rancher-token-dk5g7]: timed out waiting for the condition
17m         Normal    Pulling             pod/rancher-6bc58865f9-c97v2    Pulling image "rancher/rancher:v2.3.6"
14m         Normal    Pulled              pod/rancher-6bc58865f9-c97v2    Successfully pulled image "rancher/rancher:v2.3.6"
14m         Normal    Created             pod/rancher-6bc58865f9-c97v2    Created container rancher
14m         Normal    Started             pod/rancher-6bc58865f9-c97v2    Started container rancher
13m         Warning   Unhealthy           pod/rancher-6bc58865f9-c97v2    Readiness probe failed: Get http://10.42.0.5:80/healthz: dial tcp 10.42.0.5:80: connect: connection refused
<unknown>   Normal    Scheduled           pod/rancher-6bc58865f9-k6hgz    Successfully assigned cattle-system/rancher-6bc58865f9-k6hgz to 172.16.5.187
19m         Warning   FailedMount         pod/rancher-6bc58865f9-k6hgz    MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not found
18m         Warning   FailedMount         pod/rancher-6bc58865f9-k6hgz    Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[rancher-token-dk5g7 tls-ca-volume]: timed out waiting for the condition
17m         Normal    Pulling             pod/rancher-6bc58865f9-k6hgz    Pulling image "rancher/rancher:v2.3.6"
13m         Normal    Pulled              pod/rancher-6bc58865f9-k6hgz    Successfully pulled image "rancher/rancher:v2.3.6"
13m         Normal    Created             pod/rancher-6bc58865f9-k6hgz    Created container rancher
13m         Normal    Started             pod/rancher-6bc58865f9-k6hgz    Started container rancher
20m         Normal    SuccessfulCreate    replicaset/rancher-6bc58865f9   Created pod: rancher-6bc58865f9-6b598
20m         Normal    SuccessfulCreate    replicaset/rancher-6bc58865f9   Created pod: rancher-6bc58865f9-c97v2
20m         Normal    SuccessfulCreate    replicaset/rancher-6bc58865f9   Created pod: rancher-6bc58865f9-k6hgz
20m         Normal    ScalingReplicaSet   deployment/rancher              Scaled up replica set rancher-6bc58865f9 to 3
20m         Normal    CREATE              ingress/rancher                 Ingress cattle-system/rancher
20m         Normal    CREATE              ingress/rancher                 Ingress cattle-system/rancher
20m         Normal    CREATE              ingress/rancher                 Ingress cattle-system/rancher
13m         Normal    UPDATE              ingress/rancher                 Ingress cattle-system/rancher
13m         Normal    UPDATE              ingress/rancher                 Ingress cattle-system/rancher
13m         Normal    UPDATE              ingress/rancher                 Ingress cattle-system/rancher

kubectl -n cattle-system get ingress
NAME      HOSTS                     ADDRESS                                  PORTS     AGE
rancher   rancher-dev.pingcap.com   172.16.5.116,172.16.5.118,172.16.5.187   80, 443   21m

kubectl -n ingress-nginx logs -l app=ingress-nginx
W0331 13:54:49.288002       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 13:54:49.288051       6 controller.go:1102] Error getting SSL certificate "cattle-system/tls-rancher-ingress": local SSL certificate cattle-system/tls-rancher-ingress was not found. Using default certificate
I0331 13:55:09.738779       6 store.go:422] secret cattle-system/tls-rancher-ingress was added and it is used in ingress annotations. Parsing...
I0331 13:55:09.739779       6 backend_ssl.go:66] Adding Secret "cattle-system/tls-rancher-ingress" to the local store
W0331 13:55:09.739940       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:28.575657       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:43.347928       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:59.847238       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
I0331 14:01:18.040205       6 backend_ssl.go:58] Updating Secret "cattle-system/tls-rancher-ingress" in the local store
I0331 14:01:18.041204       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"rancher", UID:"fa6faff5-6dd3-4626-a20a-4178670ae323", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"3635", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/rancher
W0331 13:54:49.289079       6 controller.go:1102] Error getting SSL certificate "cattle-system/tls-rancher-ingress": local SSL certificate cattle-system/tls-rancher-ingress was not found. Using default certificate
I0331 13:54:49.289173       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"rancher", UID:"fa6faff5-6dd3-4626-a20a-4178670ae323", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1392", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/rancher
I0331 13:55:09.740644       6 store.go:422] secret cattle-system/tls-rancher-ingress was added and it is used in ingress annotations. Parsing...
I0331 13:55:09.741848       6 backend_ssl.go:66] Adding Secret "cattle-system/tls-rancher-ingress" to the local store
W0331 13:55:09.741979       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:28.581038       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:43.367580       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:59.849427       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
I0331 14:01:18.049047       6 backend_ssl.go:58] Updating Secret "cattle-system/tls-rancher-ingress" in the local store
I0331 14:01:18.050617       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"rancher", UID:"fa6faff5-6dd3-4626-a20a-4178670ae323", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"3635", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/rancher
W0331 13:54:49.285246       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 13:54:49.285302       6 controller.go:1102] Error getting SSL certificate "cattle-system/tls-rancher-ingress": local SSL certificate cattle-system/tls-rancher-ingress was not found. Using default certificate
I0331 13:55:09.736916       6 store.go:422] secret cattle-system/tls-rancher-ingress was added and it is used in ingress annotations. Parsing...
I0331 13:55:09.737887       6 backend_ssl.go:66] Adding Secret "cattle-system/tls-rancher-ingress" to the local store
W0331 13:55:09.738004       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:28.576267       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:43.345997       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
W0331 14:00:59.844933       6 controller.go:878] Service "cattle-system/rancher" does not have any active Endpoint.
I0331 14:01:18.040213       6 backend_ssl.go:58] Updating Secret "cattle-system/tls-rancher-ingress" in the local store
I0331 14:01:18.041196       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"rancher", UID:"fa6faff5-6dd3-4626-a20a-4178670ae323", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"3635", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/rancher

Environment information

  • Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher/stable, 2.3.6
  • Installation option (single install/HA): HA

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported): host
  • Machine type (cloud/VM/metal) and specifications (CPU/memory): VM
  • Kubernetes version (use kubectl version):
kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version (use docker version):
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false
@3AceShowHand
Copy link
Author

@superseb PTAL

@superseb
Copy link
Contributor

Can we run some additional commands to check where this is failing:

Check cluster networking between nodes:
https://rancher.com/docs/rancher/v2.x/en/troubleshooting/networking/

Check if accessing the nodes directly by faking the hostname to the node works (repeat with each host's IP for IP_OF_HOST1:

curl -k --resolve rancher.my.org:443:IP_OF_HOST1 https://rancher.my.org

Check the endpoints for rancher:

kubectl -n cattle-system get ep rancher

Check that the generated config for NGINX is containing those addresses:

for pod in $(kubectl -n ingress-nginx get pods -l app=ingress-nginx -o custom-columns=NAME:.metadata.name --no-headers); do echo $pod; kubectl -n ingress-nginx exec $pod -- curl -s --unix-socket /tmp/nginx-status-server.sock http://localhost/configuration/backends; echo;  done

Check if NGINX ingress pods can actually reach those Rancher pod addresses (this should report pong for every pod):

for rancherpod in $(kubectl -n cattle-system get ep rancher -o go-template='{{range .subsets}}{{range .addresses}}{{ .ip}}{{"\n"}}{{end}}{{end}}'); do for pod in $(kubectl -n ingress-nginx get pods -l app=ingress-nginx -o custom-columns=NAME:.metadata.name --no-headers); do echo "$pod to Rancher pod $rancherpod"; kubectl -n ingress-nginx exec $pod -- curl -sLk https://$rancherpod/ping; echo;  done; done

@3AceShowHand
Copy link
Author

3AceShowHand commented Mar 31, 2020 •
edited
Loading

@superseb Thanks for your reply, since I altered the version of tools, the result cannot be reproduced. the first comment is updated, PTAL again.
Check if overlay network is functioning correctly: no problem
Overlay network broken when using Canal/Flannel due to missing node annotations: no problem

Check if MTU is correctly configured on hosts and on peering/tunnel appliances/devices

by run `cat /sys/class/net/eth0/mtu`, all host machines return 1500, should I alter this to 1460 ?

curl -k --resolve rancher.my.org:443:IP_OF_HOST1 https://rancher.my.org

all machines return this:
{"type":"collection","links":{"self":"https://rancher-dev.pingcap.com/"},"actions":{},"pagination":{"limit":1000,"total":4},"sort":{"order":"asc","reverse":"https://rancher-dev.pingcap.com/?order=desc"},"resourceType":"apiRoot","data":[{"apiVersion":{"group":"meta.cattle.io","path":"/meta","version":"v1"},"baseType":"apiRoot","links":{"apiRoots":"https://rancher-dev.pingcap.com/meta/apiroots","root":"https://rancher-dev.pingcap.com/meta","schemas":"https://rancher-dev.pingcap.com/meta/schemas","self":"https://rancher-dev.pingcap.com/meta","subscribe":"https://rancher-dev.pingcap.com/meta/subscribe"},"type":"apiRoot"},{"apiVersion":{"group":"management.cattle.io","path":"/v3","version":"v3"},"baseType":"apiRoot","links":{"authConfigs":"https://rancher-dev.pingcap.com/v3/authconfigs","catalogs":"https://rancher-dev.pingcap.com/v3/catalogs","cloudCredentials":"https://rancher-dev.pingcap.com/v3/cloudcredentials","clusterAlertGroups":"https://rancher-dev.pingcap.com/v3/clusteralertgroups","clusterAlertRules":"https://rancher-dev.pingcap.com/v3/clusteralertrules","clusterAlerts":"https://rancher-dev.pingcap.com/v3/clusteralerts","clusterCatalogs":"https://rancher-dev.pingcap.com/v3/clustercatalogs","clusterLoggings":"https://rancher-dev.pingcap.com/v3/clusterloggings","clusterMonitorGraphs":"https://rancher-dev.pingcap.com/v3/clustermonitorgraphs","clusterRegistrationTokens":"https://rancher-dev.pingcap.com/v3/clusterregistrationtokens","clusterRoleTemplateBindings":"https://rancher-dev.pingcap.com/v3/clusterroletemplatebindings","clusterScans":"https://rancher-dev.pingcap.com/v3/clusterscans","clusterTemplateRevisions":"https://rancher-dev.pingcap.com/v3/clustertemplaterevisions","clusterTemplates":"https://rancher-dev.pingcap.com/v3/clustertemplates","clusters":"https://rancher-dev.pingcap.com/v3/clusters","composeConfigs":"https://rancher-dev.pingcap.com/v3/composeconfigs","dynamicSchemas":"https://rancher-dev.pingcap.com/v3/dynamicschemas","etcdBackups":"https://rancher-dev.pingcap.com/v3/etcdbackups","features":"https://rancher-dev.pingcap.com/v3/features","globalDnsProviders":"https://rancher-dev.pingcap.com/v3/globaldnsproviders","globalDnses":"https://rancher-dev.pingcap.com/v3/globaldnses","globalRoleBindings":"https://rancher-dev.pingcap.com/v3/globalrolebindings","globalRoles":"https://rancher-dev.pingcap.com/v3/globalroles","groupMembers":"https://rancher-dev.pingcap.com/v3/groupmembers","groups":"https://rancher-dev.pingcap.com/v3/groups","kontainerDrivers":"https://rancher-dev.pingcap.com/v3/kontainerdrivers","ldapConfigs":"https://rancher-dev.pingcap.com/v3/ldapconfigs","listenConfigs":"https://rancher-dev.pingcap.com/v3/listenconfigs","managementSecrets":"https://rancher-dev.pingcap.com/v3/managementsecrets","monitorMetrics":"https://rancher-dev.pingcap.com/v3/monitormetrics","multiClusterAppRevisions":"https://rancher-dev.pingcap.com/v3/multiclusterapprevisions","multiClusterApps":"https://rancher-dev.pingcap.com/v3/multiclusterapps","nodeDrivers":"https://rancher-dev.pingcap.com/v3/nodedrivers","nodePools":"https://rancher-dev.pingcap.com/v3/nodepools","nodeTemplates":"https://rancher-dev.pingcap.com/v3/nodetemplates","nodes":"https://rancher-dev.pingcap.com/v3/nodes","notifiers":"https://rancher-dev.pingcap.com/v3/notifiers","podSecurityPolicyTemplateProjectBindings":"https://rancher-dev.pingcap.com/v3/podsecuritypolicytemplateprojectbindings","podSecurityPolicyTemplates":"https://rancher-dev.pingcap.com/v3/podsecuritypolicytemplates","preferences":"https://rancher-dev.pingcap.com/v3/preferences","principals":"https://rancher-dev.pingcap.com/v3/principals","projectAlertGroups":"https://rancher-dev.pingcap.com/v3/projectalertgroups","projectAlertRules":"https://rancher-dev.pingcap.com/v3/projectalertrules","projectAlerts":"https://rancher-dev.pingcap.com/v3/projectalerts","projectCatalogs":"https://rancher-dev.pingcap.com/v3/projectcatalogs","projectLoggings":"https://rancher-dev.pingcap.com/v3/projectloggings","projectMonitorGraphs":"https://rancher-dev.pingcap.com/v3/projectmonitorgraphs","projectNetworkPolicies":"https://rancher-dev.pingcap.com/v3/projectnetworkpolicies","projectRoleTemplateBindings":"https://rancher-dev.pingcap.com/v3/projectroletemplatebindings","projects":"https://rancher-dev.pingcap.com/v3/projects","rkeAddons":"https://rancher-dev.pingcap.com/v3/rkeaddons","rkeK8sServiceOptions":"https://rancher-dev.pingcap.com/v3/rkek8sserviceoptions","rkeK8sSystemImages":"https://rancher-dev.pingcap.com/v3/rkek8ssystemimages","roleTemplates":"https://rancher-dev.pingcap.com/v3/roletemplates","root":"https://rancher-dev.pingcap.com/v3","self":"https://rancher-dev.pingcap.com/v3","settings":"https://rancher-dev.pingcap.com/v3/settings","subscribe":"https://rancher-dev.pingcap.com/v3/subscribe","templateVersions":"https://rancher-dev.pingcap.com/v3/templateversions","templates":"https://rancher-dev.pingcap.com/v3/templates","tokens":"https://rancher-dev.pingcap.com/v3/tokens","users":"https://rancher-dev.pingcap.com/v3/users"},"type":"apiRoot"},{"apiVersion":{"group":"cluster.cattle.io","path":"/v3/cluster","version":"v3"},"baseType":"apiRoot","links":{"apiServices":"https://rancher-dev.pingcap.com/v3/cluster/apiservices","namespaces":"https://rancher-dev.pingcap.com/v3/cluster/namespaces","persistentVolumes":"https://rancher-dev.pingcap.com/v3/cluster/persistentvolumes","root":"https://rancher-dev.pingcap.com/v3/cluster","self":"https://rancher-dev.pingcap.com/v3/cluster","storageClasses":"https://rancher-dev.pingcap.com/v3/cluster/storageclasses","subscribe":"https://rancher-dev.pingcap.com/v3/cluster/subscribe"},"type":"apiRoot"},{"apiVersion":{"group":"project.cattle.io","path":"/v3/project","version":"v3"},"baseType":"apiRoot","links":{"alertmanagers":"https://rancher-dev.pingcap.com/v3/project/alertmanagers","appRevisions":"https://rancher-dev.pingcap.com/v3/project/apprevisions","apps":"https://rancher-dev.pingcap.com/v3/project/apps","basicAuths":"https://rancher-dev.pingcap.com/v3/project/basicauths","certificates":"https://rancher-dev.pingcap.com/v3/project/certificates","configMaps":"https://rancher-dev.pingcap.com/v3/project/configmaps","cronJobs":"https://rancher-dev.pingcap.com/v3/project/cronjobs","daemonSets":"https://rancher-dev.pingcap.com/v3/project/daemonsets","deployments":"https://rancher-dev.pingcap.com/v3/project/deployments","destinationRules":"https://rancher-dev.pingcap.com/v3/project/destinationrules","dnsRecords":"https://rancher-dev.pingcap.com/v3/project/dnsrecords","dockerCredentials":"https://rancher-dev.pingcap.com/v3/project/dockercredentials","gateways":"https://rancher-dev.pingcap.com/v3/project/gateways","horizontalPodAutoscalers":"https://rancher-dev.pingcap.com/v3/project/horizontalpodautoscalers","ingresses":"https://rancher-dev.pingcap.com/v3/project/ingresses","jobs":"https://rancher-dev.pingcap.com/v3/project/jobs","namespacedBasicAuths":"https://rancher-dev.pingcap.com/v3/project/namespacedbasicauths","namespacedCertificates":"https://rancher-dev.pingcap.com/v3/project/namespacedcertificates","namespacedDockerCredentials":"https://rancher-dev.pingcap.com/v3/project/namespaceddockercredentials","namespacedSecrets":"https://rancher-dev.pingcap.com/v3/project/namespacedsecrets","namespacedServiceAccountTokens":"https://rancher-dev.pingcap.com/v3/project/namespacedserviceaccounttokens","namespacedSshAuths":"https://rancher-dev.pingcap.com/v3/project/namespacedsshauths","persistentVolumeClaims":"https://rancher-dev.pingcap.com/v3/project/persistentvolumeclaims","pipelineExecutions":"https://rancher-dev.pingcap.com/v3/project/pipelineexecutions","pipelineSettings":"https://rancher-dev.pingcap.com/v3/project/pipelinesettings","pipelines":"https://rancher-dev.pingcap.com/v3/project/pipelines","pods":"https://rancher-dev.pingcap.com/v3/project/pods","prometheusRules":"https://rancher-dev.pingcap.com/v3/project/prometheusrules","prometheuses":"https://rancher-dev.pingcap.com/v3/project/prometheuses","replicaSets":"https://rancher-dev.pingcap.com/v3/project/replicasets","replicationControllers":"https://rancher-dev.pingcap.com/v3/project/replicationcontrollers","root":"https://rancher-dev.pingcap.com/v3/project","secrets":"https://rancher-dev.pingcap.com/v3/project/secrets","self":"https://rancher-dev.pingcap.com/v3/project","serviceAccountTokens":"https://rancher-dev.pingcap.com/v3/project/serviceaccounttokens","serviceMonitors":"https://rancher-dev.pingcap.com/v3/project/servicemonitors","services":"https://rancher-dev.pingcap.com/v3/project/services","sourceCodeCredentials":"https://rancher-dev.pingcap.com/v3/project/sourcecodecredentials","sourceCodeProviderConfigs":"https://rancher-dev.pingcap.com/v3/project/sourcecodeproviderconfigs","sourceCodeProviders":"https://rancher-dev.pingcap.com/v3/project/sourcecodeproviders","sourceCodeRepositories":"https://rancher-dev.pingcap.com/v3/project/sourcecoderepositories","sshAuths":"https://rancher-dev.pingcap.com/v3/project/sshauths","statefulSets":"https://rancher-dev.pingcap.com/v3/project/statefulsets","subscribe":"https://rancher-dev.pingcap.com/v3/project/subscribe","virtualServices":"https://rancher-dev.pingcap.com/v3/project/virtualservices","workloads":"https://rancher-dev.pingcap.com/v3/project/workloads"},"type":"apiRoot"}]}

kubectl -n cattle-system get ep rancher
NAME      ENDPOINTS                                AGE
rancher   10.42.0.5:80,10.42.1.7:80,10.42.2.4:80   51m

for pod in $(kubectl -n ingress-nginx get pods -l app=ingress-nginx -o custom-columns=NAME:.metadata.name --no-headers); do echo $pod; kubectl -n ingress-nginx exec $pod -- curl -s --unix-socket /tmp/nginx-status-server.sock http://localhost/configuration/backends; echo;  done
nginx-ingress-controller-72c94
[{"name":"cattle-system-rancher-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"name":"http","protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"rancher"},"clusterIP":"10.43.212.25","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.0.5","port":"80"},{"address":"10.42.1.7","port":"80"},{"address":"10.42.2.4","port":"80"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}},{"name":"upstream-default-backend","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}],"selector":{"app":"default-http-backend"},"clusterIP":"10.43.62.76","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.2.2","port":"8080"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}}]
nginx-ingress-controller-999nz
[{"name":"cattle-system-rancher-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"name":"http","protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"rancher"},"clusterIP":"10.43.212.25","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.0.5","port":"80"},{"address":"10.42.1.7","port":"80"},{"address":"10.42.2.4","port":"80"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}},{"name":"upstream-default-backend","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}],"selector":{"app":"default-http-backend"},"clusterIP":"10.43.62.76","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.2.2","port":"8080"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}}]
nginx-ingress-controller-prnl7
[{"name":"cattle-system-rancher-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"name":"http","protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"rancher"},"clusterIP":"10.43.212.25","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.0.5","port":"80"},{"address":"10.42.1.7","port":"80"},{"address":"10.42.2.4","port":"80"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}},{"name":"upstream-default-backend","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}],"selector":{"app":"default-http-backend"},"clusterIP":"10.43.62.76","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"10.42.2.2","port":"8080"}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","cookie":""}}]

for rancherpod in $(kubectl -n cattle-system get ep rancher -o go-template='{{range .subsets}}{{range .addresses}}{{ .ip}}{{"\n"}}{{end}}{{end}}'); do for pod in $(kubectl -n ingress-nginx get pods -l app=ingress-nginx -o custom-columns=NAME:.metadata.name --no-headers); do echo "$pod to Rancher pod $rancherpod"; kubectl -n ingress-nginx exec $pod -- curl -sLk https://$rancherpod/ping; echo;  done; done
nginx-ingress-controller-72c94 to Rancher pod 10.42.0.5
pong
nginx-ingress-controller-999nz to Rancher pod 10.42.0.5
pong
nginx-ingress-controller-prnl7 to Rancher pod 10.42.0.5
pong
nginx-ingress-controller-72c94 to Rancher pod 10.42.1.7
pong
nginx-ingress-controller-999nz to Rancher pod 10.42.1.7
pong
nginx-ingress-controller-prnl7 to Rancher pod 10.42.1.7
pong
nginx-ingress-controller-72c94 to Rancher pod 10.42.2.4
pong
nginx-ingress-controller-999nz to Rancher pod 10.42.2.4
pong
nginx-ingress-controller-prnl7 to Rancher pod 10.42.2.4
pong

@3AceShowHand
Copy link
Author

which version of each tool do you recommend?

@superseb
Copy link
Contributor

Thanks for your reply, since I altered the version of tools, the result cannot be reproduced. What changed and are you still having issues? Besides the change a while back from Helm 2 to Helm 3, and a change in NGINX ingress regarding forwarding headers (https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/kubernetes-rke/#1-create-the-cluster-configuration-file), not a lot has changed in this area. To easily rule out anything in between you and the cluster nodes, does it work when you point the DNS directly at one of the nodes?

@3AceShowHand
Copy link
Author

3AceShowHand commented Mar 31, 2020 •
edited
Loading

@superseb
I have figured it out, I add the ip hostname pairs to my hosts file, the problem solved.
It's worth mentioning that, the hosts file need to be altered does not show on the official tutorial, I think this should be added.

Furthermore, according to the official tutorial's pictures, there should be 3 canel pods, but I only got 2, what is the problem? I am totally new to k8s, still a lot to learn.

@superseb
Copy link
Contributor

3 pods, one on each node and 2 containers per pod:

kube-system     canal-hql4t                               2/2     Running     0          17m
kube-system     canal-kkb5z                               2/2     Running     0          17m
kube-system     canal-tx8k2                               2/2     Running     0

Also, please describe exactly what the content of which file was before and after the change, if this change was needed on every system, no one would be able to deploy this and we do it all the time.

@3AceShowHand
Copy link
Author

3AceShowHand commented Apr 1, 2020 •
edited
Loading

3 pods, one on each node and 2 containers per pod:

kube-system     canal-hql4t                               2/2     Running     0          17m
kube-system     canal-kkb5z                               2/2     Running     0          17m
kube-system     canal-tx8k2                               2/2     Running     0

Also, please describe exactly what the content of which file was before and after the change, if this change was needed on every system, no one would be able to deploy this and we do it all the time.

I didn't change too much thing, replace tools with the following version:

rke: 1.1.0
helm: 3.1.2
kubectl: 1.17.4

once the rancher-server cluster deployed, add ip hostname pairs to the local /etc/hostname machine.

@superseb
Copy link
Contributor

superseb commented Apr 1, 2020

/etc/hostname should contain the hostname of the machine, the ip hostname format is used in /etc/hosts. If you can provide the full file before and after the modification, I can try to reproduce and see if we can check for something but so far I can't reproduce. I will close the issue as I don't see a bug here but if you can provide the requested details I can try to reproduce.

@superseb superseb closed this as completed Apr 1, 2020
@GeneralFox
Copy link

GeneralFox commented Apr 3, 2020 •
edited
Loading

I have had the same issue on a new deploy of rancher with rke + helm.
The only different things is the LB that in my case is an Haproxy 2.x but the issue was easily find with:
[root@rancher-console ~]# curl -k https://clu-rancher.swqa.tst
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

In this case with Haproxy to avoid some weid behavior in load-balancing you have to add mode tcp in the configuration.

@kgangihv kgangihv mentioned this issue Aug 3, 2020
Closed
@clayshek clayshek mentioned this issue Jan 31, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/more-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@superseb @3AceShowHand @GeneralFox