GET result: Internal Server Error on installation of worker node #174

james-liubang · 2020-05-11T06:34:44Z

Describe the bug
regarding to installation as the steps of the link,
On ESXI6.5, I have already installed service node, 1 bootstrap, 3 master node , when installed 2 worker nodes then they threw the issue as below:
[025. 447099] ignition[665] : GET result: Internal Server Error
[025. 447726] ignition[665] : GET https://api-int.lab.okd.local:22623/config/worker: attempt openshift-install doesn't support AWS GovCloud #179

i can see the console of the system on ESXI 6.5 it can't be login via ssh.

Version
https://github.com/openshift/okd/releases/download/4.4.0-0.okd-2020-04-21-163702-beta4/openshift-client-linux-4.4.0-0.okd-2020-04-21-163702-beta4.tar.gz
How reproducible
100%
Log bundle

**[core@okd4-bootstrap ~]$ oc adm must-gather
error: Missing or incomplete configuration info. Please login or point to an existing, complete config file:

Via the command-line flag --kubeconfig
Via the KUBECONFIG environment variable
In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command**

vrutkovs · 2020-05-11T07:05:18Z

See https://docs.okd.io/latest/installing/installing-troubleshooting.html#installation-bootstrap-gather_installing-troubleshooting

giatule · 2020-05-11T17:36:05Z

same issues, I tried so many times; the issue happen with compute1, compute 2 is ok

andriuss-amd · 2020-05-13T16:15:41Z

Is this happening before master nodes are bootstrapped or after? (before is a deliberate change in machine-config-operator)

james-liubang · 2020-05-15T15:54:17Z

@andriuss-xilinx it is happening after master nodes are bootstrapped .

before this , i also see another problem 'ignition[483]: Get Error: x509: certificate signed by unknown authority', seems like this:
#165

eselvam · 2020-05-17T06:58:40Z

I also face same issue while building worker node after master node. It is unable to get machine config from 22623 on bootstrap node. How to fix it.

james-liubang · 2020-05-18T07:32:31Z

Hi team, i re-install all then the above issues didn't appeared but new problem:
[core@okd4-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Logs begin at Sun 2020-05-17 06:25:46 UTC. --
May 18 07:28:47 okd4-bootstrap bootkube.sh[849]: E0518 07:28:47.736943 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:48 okd4-bootstrap bootkube.sh[849]: E0518 07:28:48.738187 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:49 okd4-bootstrap bootkube.sh[849]: E0518 07:28:49.739736 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:50 okd4-bootstrap bootkube.sh[849]: E0518 07:28:50.741369 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:51 okd4-bootstrap bootkube.sh[849]: E0518 07:28:51.742795 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:52 okd4-bootstrap bootkube.sh[849]: E0518 07:28:52.744078 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:53 okd4-bootstrap bootkube.sh[849]: E0518 07:28:53.745425 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:28:54 okd4-bootstrap bootkube.sh[849]: E0518 07:28:54.746999 1 reflector.go:153] k8s.io/client-go@v0.17.1/tools/cache/reflector.go:105: Failed to list *v1.Etcd: Get https://api-int.lab.okd.local:6443/apis/operator.openshift.io/v1/etcds?fieldSelector=metadata.name%3Dcluster&limit=500&resourceVersion=0: EOF
May 18 07:29:01 okd4-bootstrap bootkube.sh[849]: I0518 07:29:01.236780 1 waitforceo.go:67] waiting on condition EtcdRunningInCluster in etcd CR /cluster to be True.
May 18 07:29:02 okd4-bootstrap bootkube.sh[849]: I0518 07:29:02.263193 1 waitforceo.go:67] waiting on condition EtcdRunningInCluster in etcd CR /cluster to be True.

james-liubang · 2020-05-18T07:36:22Z

[admin@okd4-services ~~]$ export KUBECONFIG=~~ Error from server (ServiceUnavailable): the [admin@okd4-services ~]$ oc get nodes
NAME STATUS ROLES okd4-control-plane-1 NotReady master,worker okd4-control-plane-2 NotReady master,worker okd4-control-plane-3 NotReady master,worker [admin@okd4-services ~]$ oc get csr
NAME AGE REQUESTOR csr-2nl5k 55m system:serviceaccount:openshift- csr-2pfzr 4h26m system:node:okd4-control-plane-2 csr-2qp9n 173m system:node:okd4-control-plane-2 csr-2wpzs 4h26m system:node:okd4-control-plane-3 csr-478lr 21h system:node:okd4-control-plane-3 csr-49dpp 106m system:node:okd4-control-plane-1 csr-4zjqf 148m system:node:okd4-control-plane-3 csr-54f9n 21h system:node:okd4-control-plane-1 csr-5lfj5 4h59m system:node:okd4-control-plane-1 csr-5sj4x 4h50m system:node:okd4-control-plane-2 csr-5wp45 3h3m system:node:okd4-control-plane-1 csr-5xn9f 169m system:node:okd4-control-plane-2 csr-64mwr 121m system:node:okd4-control-plane-1 csr-662jr 169m system:node:okd4-control-plane-3 csr-666rv 143m system:node:okd4-control-plane-2 csr-699h4 70m system:serviceaccount:openshift- csr-6lnzb 21h system:serviceaccount:openshift- csr-769zr 4h58m system:node:okd4-control-plane-3 csr-7b67c 3h13m system:node:okd4-control-plane-3 csr-7fssl 148m system:node:okd4-control-plane-2 csr-7g79b 96m system:node:okd4-control-plane-2 csr-7s7vm 4h8m system:node:okd4-control-plane-3 csr-7thr7 4h58m system:node:okd4-control-plane-2 csr-7z4gr 19m system:serviceaccount:openshift- csr-8694v 5h32m system:node:okd4-control-plane-2 csr-8kmmj 106m system:node:okd4-control-plane-1 csr-96bpv 3h13m system:node:okd4-control-plane-2 csr-97l5f 3h32m system:node:okd4-control-plane-2 csr-998rd 39m system:serviceaccount:openshift- csr-bcbm7 138m system:node:okd4-control-plane-3 csr-bcwkt 86m system:serviceaccount:openshift- csr-bjwtr 24m system:serviceaccount:openshift- csr-bwkbj 8m20s system:serviceaccount:openshift- csr-clspl 3h57m system:node:okd4-control-plane-1 csr-dwnvf 4h14m system:node:okd4-control-plane-1 csr-frbkk 3h18m system:node:okd4-control-plane-1 csr-fxtl4 3h13m system:node:okd4-control-plane-2 csr-g84bv 5h34m system:node:okd4-control-plane-1 csr-gbdh5 3h19m system:node:okd4-control-plane-3 csr-gbh2w 3h34m system:node:okd4-control-plane-1 csr-gdmw2 34m system:serviceaccount:openshift- csr-gqjvb 122m system:node:okd4-control-plane-1 csr-gsnxz 3h49m system:node:okd4-control-plane-3 csr-gvl48 3h34m system:node:okd4-control-plane-3 csr-gwk6m 4h52m system:node:okd4-control-plane-2 csr-h4zkl 86m system:serviceaccount:openshift- csr-h5nzx 3h13m system:node:okd4-control-plane-2 csr-h689s 4h32m system:node:okd4-control-plane-1 csr-hv45c 21h system:serviceaccount:openshift- csr-hx6jm 23m system:serviceaccount:openshift- csr-j6rvj 5h35m system:node:okd4-control-plane-2 csr-jd6ht 3h32m system:node:okd4-control-plane-3 csr-jttcb 3h32m system:node:okd4-control-plane-2 csr-jvr6v 106m system:node:okd4-control-plane-1 csr-jzs2g 49m system:serviceaccount:openshift- csr-k7z8h 3h4m system:node:okd4-control-plane-1 csr-kd794 4h14m system:node:okd4-control-plane-1 csr-krjjs 5h32m system:node:okd4-control-plane-1 csr-l48wp 70m system:serviceaccount:openshift- csr-l6tc4 106m system:node:okd4-control-plane-3 csr-ldswc 122m system:node:okd4-control-plane-3 csr-lg2hn 173m system:node:okd4-control-plane-3 csr-lmxv7 5h13m system:node:okd4-control-plane-3 csr-lvnm7 8m57s system:serviceaccount:openshift- csr-lwpsl 5h4m system:node:okd4-control-plane-2 csr-mghkk 106m system:node:okd4-control-plane-2 csr-mxjgf 143m system:node:okd4-control-plane-3 csr-n86n2 3h13m system:node:okd4-control-plane-3 csr-nfvdn 4h26m system:node:okd4-control-plane-3 csr-nh6mq 39m system:serviceaccount:openshift- csr-nlnvv 3h40m system:node:okd4-control-plane-1 csr-nvzrd 80m system:serviceaccount:openshift- csr-nwwh9 138m system:node:okd4-control-plane-3 csr-pfsxd 142m system:node:okd4-control-plane-1 csr-pjqtw 3h51m system:node:okd4-control-plane-2 csr-pl6sc 142m system:node:okd4-control-plane-2 csr-pplfx 4h50m system:node:okd4-control-plane-3 csr-qk7xx 3h50m system:node:okd4-control-plane-1 csr-qlkc2 139m system:node:okd4-control-plane-3 csr-rc9z8 5h20m system:node:okd4-control-plane-2 csr-rlskf 3h19m system:node:okd4-control-plane-1 csr-rpnf8 65m system:serviceaccount:openshift- csr-rsftf 121m system:node:okd4-control-plane-1 csr-s2vdf 55m system:serviceaccount:openshift- csr-sts4c 142m system:node:okd4-control-plane-1 csr-t25z2 3m53s system:serviceaccount:openshift- csr-t2rvl 112m system:node:okd4-control-plane-2 csr-t7z6v 122m system:node:okd4-control-plane-3 csr-tn822 3h24m system:node:okd4-control-plane-1 csr-v45k5 139m system:node:okd4-control-plane-3 csr-v9nts 21h system:node:okd4-control-plane-2 csr-vd72r 3h50m system:node:okd4-control-plane-3 csr-vx79j 3h49m system:node:okd4-control-plane-2 csr-w5z4d 167m system:node:okd4-control-plane-1 csr-wgp4w 5h20m system:node:okd4-control-plane-1 csr-whzsh 4h52m system:node:okd4-control-plane-3 csr-wth6q 21h system:serviceaccount:openshift- csr-wvzqz 142m system:node:okd4-control-plane-3 csr-x4hhr 139m system:node:okd4-control-plane-2 csr-x5nzp 3h49m system:node:okd4-control-plane-2 csr-xvnjg 106m system:node:okd4-control-plane-3 csr-z9xp5 139m system:node:okd4-control-plane-1 csr-zbnl6 3h3m system:node:okd4-control-plane-3 csr-zggnb 54m system:serviceaccount:openshift- csr-znths 5h35m system:node:okd4-control-plane-1 csr-zrw5k 122m system:node:okd4-control-plane-2 >/install_dir/auth/kubeconfig
server is currently unable to handle the request (get users.user.openshift.io ~)
AGE VERSION
21h v1.17.1
21h v1.17.1
21h v1.17.1
CONDITION
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Pending
Approved,Issued
Approved,Issued
Pending
Pending
Approved,Issued
Approved,Issued
Approved,Issued
Pending
Pending
Approved,Issued
Pending
Pending
machine-config-operator:node-bootstrapper Pending
machine-config-operator:node-bootstrapper Approved,Issued
Pending
Pending
Pending
Approved,Issued
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
Pending
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Pending
machine-config-operator:node-bootstrapper Pending
machine-config-operator:node-bootstrapper Pending
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
Pending
Pending
Approved,Issued
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Pending
Approved,Issued
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Approved,Issued
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Pending
Pending
Pending
Pending
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Pending
Pending
Pending
Pending
machine-config-operator:node-bootstrapper Pending
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Pending
Pending
Approved,Issued
Pending
Pending
Pending
Pending
Approved,Issued
Pending
machine-config-operator:node-bootstrapper Pending
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Pending
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Approved,Issued
Approved,Issued
Pending
Approved,Issued
Approved,Issued
Pending
Pending
Approved,Issued
Approved,Issued
machine-config-operator:node-bootstrapper Approved,Issued
Pending
Pending
Approved,Issued
Approved,Issued
Pending
Approved,Issued
machine-config-operator:node-bootstrapper Pending
Approved,Issued
Pending

thomasmeeus · 2020-05-18T18:47:14Z

same issue here; Bootstrapping master nodes goes fine. Worker nodes get an "Internal server errror" when fetching their config from the internal api. 100% reproducable with 4.4.0-0.okd-2020-05-18-142413

thomasmeeus · 2020-05-18T20:25:22Z

@vrutkovs log bundle can be downloaded over here: https://sector7g.be/log-bundle-20200518213832.tar.gz

Fyi: I've deployed similar clusters with the redhat openshift flavor without issues, so I'm quite experienced with the deployment procedure, but it could be something very stupid I'm doing wrong

some additional info:

$loadbalancer_ip:6443 & $loadbalancer_ip:22623 are "green" in the loadbalancer for all master nodes (so listening on the tcp port). The 2 workers stay red. Both workers have the "Internetal server error" on the vmware console.
it seems that after a long time (+1 hour) of getting the "internal server error", it seems the worker nodes seem to be able to pull in their config. It takes at least +100 failed iterations. Although, it seems the cluster never becomes a healthy state.

$ .tmp/./openshift-install wait-for bootstrap-complete --dir .tmp/config/
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp4.cegeka.io:6443...
INFO API v1.17.1 up
INFO Waiting up to 40m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources

$ oc login https://api.ocp4.cegeka.io:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): yes

error: couldn't get https://api.ocp4.cegeka.io:6443/.well-known/oauth-authorization-server: unexpected response status 404

curl -k -v https://api.ocp4.cegeka.io:6443
* Rebuilt URL to: https://api.ocp4.cegeka.io:6443/
*   Trying 172.29.49.101...
* TCP_NODELAY set
* Connected to api.ocp4.cegeka.io (172.29.49.101) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.ocp4.cegeka.io
*  start date: May 18 19:31:58 2020 GMT
*  expire date: Jun 17 19:31:59 2020 GMT
*  issuer: OU=openshift; CN=kube-apiserver-lb-signer
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7febef00b400)
> GET / HTTP/2
> Host: api.ocp4.cegeka.io:6443
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 403
< audit-id: 8f1c62b6-edd5-4c53-8bc6-8345cc1f48e3
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 233
< date: Mon, 18 May 2020 20:15:01 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
* Connection #0 to host api.ocp4.cegeka.io left intact

$ export KUBEADMIN=auth/kubeconfig
$ oc get nodes
error: You must be logged in to the server (Unauthorized)

vrutkovs · 2020-05-19T07:06:27Z

INFO It is now safe to remove the bootstrap resources

Is the bootstrap node removed after that message?

thomasmeeus · 2020-05-19T08:27:14Z

Nope, the bootstrap node remains online. I know by experience that this is fine for Redhat Openshift.
Should it get removed for OKD? Maybe because it doesn't server the compute-configuration and the loadbalancer still points to it?

vrutkovs · 2020-05-19T08:37:29Z

Nope, the bootstrap node remains online.

Why? Bootstrap's MCS won't serve files to any node except masters - openshift/machine-config-operator@c894879

thomasmeeus · 2020-05-19T11:06:36Z

Cool. I didn't know it came that close, but removing the bootstrap node indeed fixed it.
Maybe the log line could be clarified a bit, cause now I got the impression that it wasn't mandatory to remove the boostrap node.

Many thanks!
I'm not the owner of this issue, but I guess it's solved now

vrutkovs · 2020-05-19T13:50:05Z

No must-gather for more than a week, closing.

Please refrain from "same here" and "+1", file different issues

eselvam · 2020-05-19T15:57:44Z

so, after master node bootstraped, do we need to remove bootstrap node before we bring up worker node?

Kindly clarify.

rainbowjose · 2020-07-07T00:51:57Z

Its is normal. You just need to wait until installing complete. Patience.

…okd-project#174)

vrutkovs added platform/vsphere triage/needs-information Indicates an issue needs more information in order to work on it. labels May 11, 2020

vrutkovs closed this as completed May 19, 2020

jam01 mentioned this issue Jun 6, 2020

Bootstrap not tearing down, workers can't retrieve config #209

Closed

binnes pushed a commit to binnes/okd that referenced this issue Jul 11, 2023

Update 2021-03-22-recap-okd-testing-deployment-workshop.html.markdown (…

a128969

…okd-project#174)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GET result: Internal Server Error on installation of worker node #174

GET result: Internal Server Error on installation of worker node #174

james-liubang commented May 11, 2020 •

edited

vrutkovs commented May 11, 2020

giatule commented May 11, 2020

andriuss-amd commented May 13, 2020

james-liubang commented May 15, 2020

eselvam commented May 17, 2020

james-liubang commented May 18, 2020

james-liubang commented May 18, 2020

thomasmeeus commented May 18, 2020

thomasmeeus commented May 18, 2020

vrutkovs commented May 19, 2020

thomasmeeus commented May 19, 2020

vrutkovs commented May 19, 2020

thomasmeeus commented May 19, 2020

vrutkovs commented May 19, 2020

eselvam commented May 19, 2020 •

edited

rainbowjose commented Jul 7, 2020

GET result: Internal Server Error on installation of worker node #174

GET result: Internal Server Error on installation of worker node #174

Comments

james-liubang commented May 11, 2020 • edited

vrutkovs commented May 11, 2020

giatule commented May 11, 2020

andriuss-amd commented May 13, 2020

james-liubang commented May 15, 2020

eselvam commented May 17, 2020

james-liubang commented May 18, 2020

james-liubang commented May 18, 2020

thomasmeeus commented May 18, 2020

thomasmeeus commented May 18, 2020

vrutkovs commented May 19, 2020

thomasmeeus commented May 19, 2020

vrutkovs commented May 19, 2020

thomasmeeus commented May 19, 2020

vrutkovs commented May 19, 2020

eselvam commented May 19, 2020 • edited

rainbowjose commented Jul 7, 2020

james-liubang commented May 11, 2020 •

edited

eselvam commented May 19, 2020 •

edited