openshift-install failed to install on vsphere and API is not up

[skywidetech]

Version

$ openshift-install version
bin/openshift-install unreleased-master-3930-g8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529
built from commit 8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529
release image registry.svc.ci.openshift.org/origin/release:4.5

Platform:

vsphere

IPI

What happened?

I am running the latest version of installer on vsphere. I can see that the VMs are created for master, worker etc, but the API is not responding. 192.168.10.31 and 192.168.10.32 are 2 unused internal IP in the LAN. the 2 IPs are updated in the DNS with reverse lookup also
192.168.10.31 <-> api.dev.mycompany.com
192.168.10.32 <-> *.apps.dev.mycompany.com

[root@localhost installer]# hack/build.sh
+ minimum_go_version=1.14
++ go version
++ cut -d ' ' -f 3
+ current_go_version=go1.15.2
++ version 1.15.2
++ IFS=.
++ printf '%03d%03d%03d\n' 1 15 2
++ unset IFS
++ version 1.14
++ IFS=.
++ printf '%03d%03d%03d\n' 1 14
++ unset IFS
+ '[' 001015002 -lt 001014000 ']'
+ MODE=release
++ git rev-parse --verify 'HEAD^{commit}'
+ GIT_COMMIT=8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529
++ git describe --always --abbrev=40 --dirty
+ GIT_TAG=unreleased-master-3930-g8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529
+ GOFLAGS=-mod=vendor
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-3930-g8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529 -X github.com/openshift/installer/pkg/version.Commit=8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529'
+ TAGS=
+ OUTPUT=bin/openshift-install
+ export CGO_ENABLED=0
+ CGO_ENABLED=0
+ case "${MODE}" in
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-3930-g8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529 -X github.com/openshift/installer/pkg/version.Commit=8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529 -s -w'
+ TAGS=' release'
+ test '' '!=' y
+ go generate ./data
writing assets_vfsdata.go
+ echo ' release'
+ grep -q libvirt
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-3930-g8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529 -X github.com/openshift/installer/pkg/version.Commit=8d9d7cb16bd681f5ff5ff1b22305d3a6d5466529 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
[root@localhost installer]# bin/openshift-install create cluster
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform vsphere
? vCenter 192.168.10.15
? Username administrator@vsphere.local
? Password [? for help] *********
                        INFO Connecting to vCenter 192.168.10.15          
INFO Defaulting to only available datacenter: mycompany
INFO Defaulting to only available cluster: dev    
? Default Datastore mycompany-f-datastore1
? Network VM Network
? Virtual IP Address for API 192.168.10.31
? Virtual IP Address for Ingress 192.168.10.32
? Base Domain mycompany.com
? Cluster Name dev
? Pull Secret [? for help] ****************************************************************************************************************
                           INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.7/47.82.202010211043-0/x86_64/rhcos-47.82.202010211043-0-vmware.x86_64.ova?sha256=4d11143504f89b9b7ebb2a2962dc3ace6a9f92c2bdf3307fb1ba615dcfedbe43' 
INFO Creating infrastructure resources...         
INFO Waiting up to 20m0s for the Kubernetes API at https://api.dev.mycompany.com:6443... 
ERROR Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: Get "https://api.dev.mycompany.com:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp 192.168.10.31:6443: connect: no route to host 
ERROR Attempted to gather debug logs after installation failure: failed to get bootstrap and control plane host addresses from "terraform.tfstate": failed to lookup bootstrap ipv4 address: Post "https://192.168.10.15/sdk": context deadline exceeded 
FATAL Bootstrap failed to complete: failed waiting for Kubernetes API: Get "https://api.dev.mycompany.com:6443/version?timeout=32s": dial tcp 192.168.10.31:6443: connect: no route to host 

[staebler]

Please check that the bootstrap machine is running and accessible. The bootstrap machine should be the first one to take control of the API VIP.

Also, there appears to be a problem communicating with the vCenter at the end.

failed to lookup bootstrap ipv4 address: Post "https://192.168.10.15/sdk"

[skywidetech]

The bootstrap is up, but it comes up with this error:

pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 
msg='unit=zincati comm="systemd" exe="/usr/lib/systemd/systemd" host=? addr=? terminal=? res=success'

https://192.168.10.15/sdk should not be accessible according to vmware.
See here: https://kb.vmware.com/s/article/1003218

[staebler]

https://192.168.10.15/sdk should not be accessible according to vmware.
See here: https://kb.vmware.com/s/article/1003218

I think that you are misreading that article. That article is stating that sdk URL is not serving a web page, so it is not "accessible" from a web browser. It should be accessible generally, however.

[skywidetech]

but if I visit this URL "https://192.168.10.15/sdk/vimService.wsdl" according to the article, it works just fine as below

<!-- 
   Copyright 2005-2018 VMware, Inc.  All rights reserved.
 -->
<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:interface="urn:vim25" targetNamespace="urn:vim25Service">
<import location="vim.wsdl" namespace="urn:vim25"/>
<service name="VimService">
<port binding="interface:VimBinding" name="VimPort">
<soap:address location="https://localhost/sdk/vimService"/>
</port>
</service>
</definitions>

[skywidetech]

The following is a snippet from "vendor/github.com/vmware/govmomi/vim25/soap/client.go"
It sets the default path to "/sdk". Should it be "/sdk/vimService" instead? according to vmware doc, the correct endpoint address of the SOAP call is "/sdk/vimService". I am not sure if this is a bug

// ParseURL is wrapper around url.Parse, where Scheme defaults to "https" and Path defaults to "/sdk"
func ParseURL(s string) (*url.URL, error) {
	var err error
	var u *url.URL

	if s != "" {
		// Default the scheme to https
		if !schemeMatch.MatchString(s) {
			s = "https://" + s
		}

		u, err = url.Parse(s)
		if err != nil {
			return nil, err
		}

		// Default the path to /sdk
		if u.Path == "" {
			u.Path = "/sdk"
		}

		if u.User == nil {
			u.User = url.UserPassword("", "")
		}
	}

	return u, nil
}

[vrutkovs]

Please use https://github.com/openshift/okd for OKD installer issues

Dupe of okd-project/okd#394

/close

[openshift-ci-robot]

@vrutkovs: Closing this issue.

In response to this:

Please use https://github.com/openshift/okd for OKD installer issues

Dupe of okd-project/okd#394

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.