openshift-install not creating the worker vm using IPI

[bobby0724]

Describe the bug
I am trying to deploy a new cluster into my vsphere infrastructure using IPI
Bootstrap process completes but the workers are never created

Version

openshift-install 4.5.0-0.okd-2020-10-15-235428
IPI installation
How reproducible

i have try 10 times bootstrap completes but the workers are never created

this is my install-config.yml
apiVersion: v1
baseDomain: okd.local
metadata:
name: lab

compute:

• hyperthreading: Enabled
  name: worker
  replicas: 3
  platform:
  vsphere:
  cpus: 4
  coresPerSocket: 4
  memoryMB: 16384
  osDisk:
  diskSizeGB: 120

controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
platform:
vsphere:
cpus: 4
coresPerSocket: 4
memoryMB: 16384
osDisk:
diskSizeGB: 120

networking:
clusterNetworks:

• cidr: 10.254.0.0/16
  hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
• 172.30.0.0/16

Log bundle

DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 86% complet

[fortinj66]

No workers are created in VMWare or in the cluster?

[bobby0724]

No workers are created in VMWare or in the cluster?

no vm workers are created, only the masters, i also notice that the ingress operator never startup

[fortinj66]

what does the VMWare terminal indicate logwise for the masters?

can you run oc get nodes and oc get events

[bobby0724]

what does the VMWare terminal indicate logwise for the masters?

can you run oc get nodes and oc get events

give me 10 min i will ran the installation and i can get you all the info

[bobby0724]

Here is the log of the cluster creation

You can see the vms created on this image
https://flic.kr/p/2ka6e65

cluster creation log

DEBUG vsphereprivate_import_ova.import: Creating...
DEBUG vsphereprivate_import_ova.import: Still creating... [10s elapsed]
DEBUG vsphereprivate_import_ova.import: Still creating... [20s elapsed]
DEBUG vsphereprivate_import_ova.import: Creation complete after 23s [id=vm-362769]
DEBUG data.vsphere_virtual_machine.template: Refreshing state...
DEBUG module.master.vsphere_virtual_machine.vm[2]: Creating...
DEBUG module.bootstrap.vsphere_virtual_machine.vm: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[1]: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[0]: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[0]: Creation complete after 2s [id=42093637-8ddf-997c-9649-294b67179db6]
DEBUG module.master.vsphere_virtual_machine.vm[2]: Creation complete after 7s [id=420953d6-89f4-38e9-dd92-0447803772e5]
DEBUG module.master.vsphere_virtual_machine.vm[1]: Creation complete after 8s [id=42091a7d-cd34-c6be-342a-15ffb79ccdfc]
DEBUG module.bootstrap.vsphere_virtual_machine.vm: Creation complete after 9s [id=420925b7-151f-c43c-2221-2c851e35855b]
DEBUG
DEBUG Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443...
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 86% complete, waiting on authentication, console, ingress, kube-storage-version-migrator, monitoring

$ oc get nodes
NAME STATUS ROLES AGE VERSION
lab-4dgs7-master-0 Ready master 9m12s v1.18.3
lab-4dgs7-master-1 Ready master 9m30s v1.18.3
lab-4dgs7-master-2 Ready master 9m8s v1.18.3

$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
5m52s Normal LeaderElection configmap localhost_98641d9a-e659-4680-9baa-1a536a4c93a9 stopped leading
5m51s Normal LeaderElection endpoints localhost_976bcdba-2098-4571-b664-0b3ec6760a35 stopped leading
4m23s Normal TerminationStart namespace/kube-system Received signal to terminate, becoming unready, but keeping serving
4m23s Normal TerminationPreShutdownHooksFinished namespace/kube-system All pre-shutdown hooks have been finished
9m14s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
2m29s Normal NodeDone node/lab-4dgs7-master-0 Setting node lab-4dgs7-master-0, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
9m29s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
2m29s Normal NodeDone node/lab-4dgs7-master-1 Setting node lab-4dgs7-master-1, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
9m9s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
105s Normal NodeDone node/lab-4dgs7-master-2 Setting node lab-4dgs7-master-2, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
4m21s Normal Status upgrade clusteroperator/machine-api Progressing towards operator: 4.5.0-0.okd-2020-10-15-235428
7m43s Normal OperatorVersionChanged /machine-config clusteroperator/machine-config-operator started a version change from [] to [{operator 4.5.0-0.okd-2020-10-15-235428}]
100s Normal OperatorVersionChanged /machine-config clusteroperator/machine-config-operator version changed from [] to [{operator 4.5.0-0.okd-2020-10-15-235428}]

$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication Unknown Unknown True 7m52s
cloud-credential 4.5.0-0.okd-2020-10-15-235428 True False False 10m
cluster-autoscaler 4.5.0-0.okd-2020-10-15-235428 True False False 3m39s
config-operator 4.5.0-0.okd-2020-10-15-235428 True False False 3m54s
console 4.5.0-0.okd-2020-10-15-235428 Unknown True False 3m49s
csi-snapshot-controller 4.5.0-0.okd-2020-10-15-235428 True False False 4m17s
dns 4.5.0-0.okd-2020-10-15-235428 True False False 7m9s
etcd 4.5.0-0.okd-2020-10-15-235428 True False False 6m14s
image-registry 4.5.0-0.okd-2020-10-15-235428 True False False 4m24s
ingress False True True 3m38s
insights 4.5.0-0.okd-2020-10-15-235428 True False False 4m28s
kube-apiserver 4.5.0-0.okd-2020-10-15-235428 True True True 5m28s
kube-controller-manager 4.5.0-0.okd-2020-10-15-235428 True True False 6m14s
kube-scheduler 4.5.0-0.okd-2020-10-15-235428 True True False 4m31s
kube-storage-version-migrator 4.5.0-0.okd-2020-10-15-235428 False False False 7m53s
machine-api 4.5.0-0.okd-2020-10-15-235428 True False False 96s
machine-approver 4.5.0-0.okd-2020-10-15-235428 True False False 4m18s
machine-config 4.5.0-0.okd-2020-10-15-235428 True False False 2m8s
marketplace 4.5.0-0.okd-2020-10-15-235428 True False False 3m57s
monitoring Unknown True Unknown 4m56s
network 4.5.0-0.okd-2020-10-15-235428 True False False 8m12s
node-tuning 4.5.0-0.okd-2020-10-15-235428 True False False 7m59s
openshift-apiserver 4.5.0-0.okd-2020-10-15-235428 True False False 4m35s
openshift-controller-manager 4.5.0-0.okd-2020-10-15-235428 True False False 4m5s
openshift-samples 4.5.0-0.okd-2020-10-15-235428 True False False 2m55s
operator-lifecycle-manager 4.5.0-0.okd-2020-10-15-235428 True False False 7m29s
operator-lifecycle-manager-catalog 4.5.0-0.okd-2020-10-15-235428 True False False 7m31s
operator-lifecycle-manager-packageserver 4.5.0-0.okd-2020-10-15-235428 True False False 4m38s
service-ca 4.5.0-0.okd-2020-10-15-235428 True False False 7m54s
storage 4.5.0-0.okd-2020-10-15-235428 True False False 4m27s

[fortinj66]

It doesn't looks like the bootstrap process has finished... Nodes get created after the bootstrap process is complete...

try running ./openshift-install gather bootstrap --dir=<installation_directory> and post the results...

(https://docs.okd.io/latest/installing/installing-troubleshooting.html#installation-bootstrap-gather_installing-troubleshooting)

[fortinj66]

also, if possible, you may want to make sure all the nodes are running on different VMWare hosts, or as much as possible... I've had issues were OKD hosts saturated the VMWare host CPU and it caused issues...

[bobby0724]

It doesn't looks like the bootstrap process has finished... Nodes get created after the bootstrap process is complete...

try running ./openshift-install gather bootstrap --dir=<installation_directory> and post the results...

(https://docs.okd.io/latest/installing/installing-troubleshooting.html#installation-bootstrap-gather_installing-troubleshooting)

it does finish look

DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443...
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.okd.local:6443/version?timeout=32s: dial tcp 192.168.1.201:6443: connect: connection refused
INFO API v1.18.3 up
INFO Waiting up to 40m0s for bootstrapping to complete... DEBUG Bootstrap status: complete
INFO Destroying the bootstrap resources...
DEBUG Symlinking plugin terraform-provider-aws src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-aws"
DEBUG Symlinking plugin terraform-provider-google src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-google"
DEBUG Symlinking plugin terraform-provider-openstack src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-openstack"
DEBUG Symlinking plugin terraform-provider-vsphere src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-vsphere"
DEBUG Symlinking plugin terraform-provider-random src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-random"
DEBUG Symlinking plugin terraform-provider-vsphereprivate src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-vsphereprivate"
DEBUG Symlinking plugin terraform-provider-azurerm src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-azurerm"
DEBUG Symlinking plugin terraform-provider-azureprivatedns src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-azureprivatedns"
DEBUG Symlinking plugin terraform-provider-ignition src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-ignition"
DEBUG Symlinking plugin terraform-provider-local src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-local"
DEBUG Symlinking plugin terraform-provider-ovirt src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-ovirt"
DEBUG Initializing modules...
DEBUG - bootstrap in ../../../tmp/openshift-install-612627491/bootstrap
DEBUG - master in ../../../tmp/openshift-install-612627491/master

[fortinj66]

The image you sent and the logs didn't show that completing... Bootstrap node was still there...

what vmware hosts are created now?

[bobby0724]

The image you sent and the logs didn't show that completing... Bootstrap node was still there...

what vmware hosts are created now?

No, that is the issue I am having bootstrap completes but workers are never created, the cluster provisioning never completes

[fortinj66]

can you run oc adm must-gather

and another shot of your currently created VMWare nodes?

and check oc get csr

[bobby0724]

can you run oc adm must-gather

and another shot of your currently created VMWare nodes?

and check oc get csr

$ oc get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-2mvx2 8m58s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-1 Approved,Issued
csr-8qvkh 9m17s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-x6b4s 8m38s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-2 Approved,Issued
csr-xxx7v 9m3s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-zbt4g 9m1s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-zp5l2 8m40s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-0 Approved,Issued

the output of the gather command is to big to paste it here what are you looking for
bootstrap vm is deleted every time

[fortinj66]

Do you have dropbox or similar you can put that logfile in?

[bobby0724]

Do you have dropbox or similar you can put that logfile in?

sure

https://www.dropbox.com/s/w09xr5js43btt29/gather.txt?dl=0