libvirt openshift-installer does not work

[th3penguinwhisperer]

I've tried OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE env to test OCP...
with registry.svc.ci.openshift.org/ocp/release:4.3 but got a different issue now (even earlier):

• The webservice on the bootstrap node never comes up.
• The bootkube doesn't start because crio-configure doesn't start because release-image.service doesn't start.
• Running /usr/local/bin/release-image-download.sh shows the issue:

Pulling registry.svc.ci.openshift.org/ocp/release:4.3...
Error: error pulling image "registry.svc.ci.openshift.org/ocp/release:4.3": unable to pull registry.svc.ci.openshift.org/ocp/release:4.3: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/ocp/release:4.3: Error reading manifest 4.3 in registry.svc.ci.openshift.org/ocp/release: unauthorized: authentication required
Pull failed. Retrying registry.svc.ci.openshift.org/ocp/release:4.3...

• I've checked the config.json and there's no auth token for this registry.
• So the master doesn't even get to the point where it gets the config.

I assume this is some kind of a workaround? I just want to basically install the same as would be on VMware vSphere so it feels odd having to use a different repo.
Am I blocked from using release-42 release-43 ... and can only use master?

If this is not a workaround it would be good the docs are updated (https://github.com/openshift/installer/blob/master/docs/dev/libvirt/README.md) as there's no way to figure this out.

/label platform/libvirt

Originally posted by @th3penguinwhisperer in #3030 (comment)

[abhinavdahiya]

I've tried OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE env to test OCP...
with registry.svc.ci.openshift.org/ocp/release:4.3 but got a different issue now (even earlier):

If you are trying to test OCP, we don't support libvirt backend. it is mean only for developement.
and Red hat people have access to registry.svc.ci.openshift.org/ocp/release:4.3 with a special pull secret.

So if you want to test OCP, use the nightly builds from https://openshift-release.svc.ci.openshift.org/#4.3.0-0.nightly
That should be available to OCP users.

And if you want to try libvirt and OCP, you can use one of the official release images, can be found here https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/release.txt

[th3penguinwhisperer]

Thanks for the assistance.

In the mean time I had used the image that was mentioned from quay.io here: https://openshift-release.svc.ci.openshift.org/releasestream/4-stable/release/4.3.0 which seems to be the same on your second link.

The master and node are now deploying however it finishes with an error.
FATAL failed to initialize the cluster: Working towards 4.3.0: 98% complete, waiting on authentication, console, monitoring, openshift-samples
I believe this is just because my system is slow. I can connect to the console and log in.

Any chance the images to image override can be documented as well?

[thdevelthreesixtiy]

@th3penguinwhisperer
I have the same issue (#2927) and tried get the workaround using OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE in place.

env OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_64

TAGS=libvirt hack/build.sh

But running the installation with this build does not result in different images beeing used by the installer. It just uses the original images.

Is it correct to set the var before compilation of the installer the way I did it?

[thdevelthreesixtiy]

I managed it in the meantime by myself.
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_64
has to be set berfore running the installer.
A proper pull secret from RedHat has to be entered for usage of OCP images.

[sallyom]

I have been successfully installing libvirt, however, I've found that I have to give installs a few extra minutes beyond the install timeout:
If you:
openshift-install create cluster || true then upon timing out, run a loop to wait for all COs to become available, you'll find the installs proceed to a healthy cluster.

Installer timeout can be extended for libvirt installs? Or, need to investigate what, with libvirt, is causing installs to take so long.

openshift/release#7568

[th3penguinwhisperer]

So I got busy with this again... Hoped I would run into less troubles this time around which was obviously silly of me :)

I used the 4.3.25 override and built openshift-installer from branch release-4.3.

...
* Could not update servicemonitor "openshift-kube-scheduler-operator/kube-scheduler-operator" (475 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (169 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-machine-api/machine-api-operator" (480 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-machine-config-operator/machine-config-daemon" (482 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-operator-lifecycle-manager/olm-operator" (484 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator" (441 of 498): the server does not recognize this resource, check extension API servers
* Could not update servicemonitor "openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator" (445 of 498): the server does not recognize this resource, check extension API servers 
DEBUG Still waiting for the cluster to initialize: Working towards 4.3.25: 73% complete 
DEBUG Still waiting for the cluster to initialize: Working towards 4.3.25: 74% complete 
DEBUG Still waiting for the cluster to initialize: Working towards 4.3.25: 81% complete 
DEBUG Still waiting for the cluster to initialize: Working towards 4.3.25: 82% complete 
ERROR Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: Get https://api.oc4.pb.local:6443/apis/config.openshift.io/v1/clusteroperators: dial tcp 192.168.126.11:6443: connect: no route to host 
FATAL failed to initialize the cluster: Working towards 4.3.25: 82% complete

Then I do a wait-for install-complete:

...
DEBUG Still waiting for the cluster to initialize: Working towards 4.3.25: 99% complete 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring 
ERROR Cluster operator authentication Degraded is True with RouteStatusDegradedFailedHost: RouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.oc4.pb.local: [] 
INFO Cluster operator authentication Progressing is Unknown with NoData:  
INFO Cluster operator authentication Available is Unknown with NoData:  
INFO Cluster operator console Progressing is True with RouteSync_FailedHost: RouteSyncProgressing: route is not available at canonical host [] 
INFO Cluster operator console Available is Unknown with NoData:  
INFO Cluster operator image-registry Available is False with NoReplicasAvailable: The deployment does not have available replicas 
INFO Cluster operator image-registry Progressing is True with DeploymentNotCompleted: The deployment has not completed 
ERROR Cluster operator ingress Degraded is True with IngressControllersDegraded: Some ingresscontrollers are degraded: default 
INFO Cluster operator ingress Progressing is True with Reconciling: Not all ingress controllers are available.
Moving to release version "4.3.25".
Moving to ingress-controller image version "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be0f41b0aa2203a8a5a929af1d34bb8ab7990c698ea7ac8a78b84ec92d1afebd". 
INFO Cluster operator ingress Available is False with IngressUnavailable: Not all ingress controllers are available. 
INFO Cluster operator insights Disabled is False with :  
INFO Cluster operator monitoring Available is False with :  
INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack. 
ERROR Cluster operator monitoring Degraded is True with UpdatingAlertmanagerFailed: Failed to rollout the stack. Error: running task Updating Alertmanager failed: waiting for Alertmanager Route to become ready failed: waiting for RouteReady of alertmanager-main: no status available for alertmanager-main 
FATAL failed to initialize the cluster: Some cluster operators are still updating: authentication, console, image-registry, ingress, monitoring

At this point the bootstrap VM is already deleted and I only have a "master" VM left. No worker/node. My setup is very simple since I'm trying to get it deployed first and then I'll scale or do modifications. 1 master/1 worker.
When looking up those errors I got to a RH solutions article which claimed this might be an issue with the provisioning being too small. So I tried with settings those TF_VAR_libvirt_master_memory that I found somewhere in a git commit. That seems to be broken and updating the tf vars file at this point is not possible. So I had to edit the source code and rebuild openshift-installer after which I got the same problem (this time the CPU and memory was 16GB and 8vCPU).

Also /var/lib/libvirt/openshift-images ran out of space since it's apparently not cleaned up properly.

I did set the wildcard dns entry to the master IP which is probably incorrect but since there's no other OC4 VMs it's the only thing I could do.
At what point should the worker be deployed? After the bootstrap is destroyed and the master is up and operational?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale