Unable to bundle-upgrade

[J0zi]

Bug Report

What did you do?

operator-sdk run bundle quay.io/community-operators-pipeline/flux:v0.25.2

operator-sdk run bundle-upgrade quay.io/operator_testing/flux:testing0.25.3 

What did you expect to see?

Successful upgrade

What did you see instead? Under which circumstances?

operator-sdk run bundle-upgrade quay.io/operator_testing/flux:testing0.25.3       
INFO[0001] Found existing subscription with name flux-v0-25-2-sub and namespace default 
INFO[0001] Found existing catalog source with name flux-catalog and namespace default 
INFO[0009] Successfully created registry pod: quay-io-operator-testing-flux-testing0-25-3 
INFO[0009] Updated catalog source flux-catalog with address and annotations 
INFO[0009] Deleted previous registry pod with name "quay-io-community-operators-pipeline-flux-v0-25-2" 
INFO[0048] Approved InstallPlan install-5xp65 for the Subscription: flux-v0-25-2-sub 
INFO[0048] Waiting for ClusterServiceVersion "default/flux.v0.25.3" to reach 'Succeeded' phase 
INFO[0048]   Waiting for ClusterServiceVersion "default/flux.v0.25.3" to appear 
FATA[0120] Failed to run bundle upgrade: error waiting for CSV to install: timed out waiting for the condition 

kubectl logs catalog-operator-5c4997c789-47bxt -n olm

time="2022-03-22T11:32:37Z" level=info msg="checking flux.v0.25.2"
time="2022-03-22T11:32:37Z" level=info msg="checking packageserver"
time="2022-03-22T11:32:37Z" level=info msg="checking packageserver"
time="2022-03-22T11:32:37Z" level=info msg="checking flux.v0.25.2"
time="2022-03-22T11:32:42Z" level=info msg="Processing olmConfig"
time="2022-03-22T11:32:43Z" level=info msg="Processing olmConfig"
time="2022-03-22T11:32:47Z" level=info msg="checking flux.v0.25.2"
time="2022-03-22T11:32:47Z" level=info msg="checking packageserver"
time="2022-03-22T11:32:47Z" level=info msg="checking flux.v0.25.2"
time="2022-03-22T11:32:52Z" level=info msg="Processing olmConfig"
time="2022-03-22T11:32:52Z" level=info msg="Processing olmConfig"
time="2022-03-22T11:33:30Z" level=info msg="checking packageserver"
time="2022-03-22T11:33:30Z" level=info msg="checking flux.v0.25.2"
time="2022-03-22T11:33:35Z" level=info msg="Processing olmConfig"
{"level":"error","ts":1647948830.2065864,"logger":"controllers.operatorcondition","msg":"Unable to find operatorcondition","request":"default/flux.v0.25.3","error":"OperatorCondition.operators.coreos.com \"flux.v0.25.3\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1647948830.2067282,"logger":"controller.operatorcondition","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"OperatorCondition","name":"flux.v0.25.3","namespace":"default","error":"OperatorCondition.operators.coreos.com \"flux.v0.25.3\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

Environment

Kubernetes cluster type:

kind cluster with k8s 1.21

Additional information

You replicate easily by just 2 commands.

operator-sdk run bundle quay.io/community-operators-pipeline/flux:v0.25.2
operator-sdk run bundle-upgrade quay.io/operator_testing/flux:testing0.25.3 

or

operator-sdk run bundle quay.io/community-operators-pipeline/apicurito:v1.0.2
operator-sdk run bundle-upgrade quay.io/operator_testing/apicurito:testing-apicurito.v1.0.3

It is blocking us delivering a new feature - Operator upgrade testing.

[camilamacedo86]

It shows the same scenario of the following issues:

• FATA[0120] Failed to run bundle: install plan is not available for the subscription memcached-operator-v0-0-1-sub: timed out waiting for the condition #5413
• Failed to run bundle: install plan is not available for the subscription #5410

Additionally, it shows that we need to improve the run bundle to help us know why is not possible to get the operator conditional: #3691 (comment)

It might be the same scenario of:
#4650

@J0zi the run bundle does not work if the bundle is not published in a public registry.
Could you try: #4650 (comment). Is not this the case: https://master.sdk.operatorframework.io/docs/olm-integration/cli-overview/#private-bundle-and-catalog-image-registries?

c/c @rashmigottipati

[rashmigottipati]

I was able to reproduce this issue. The registry pod and other resources are being successfully for the image, so it might not be an issue related to pulling a private bundle image. The catalog source that's created seems healthy as well. The bundle image is hosted on a public registry and is also pull-able locally so secrets don't have to be provided as well.

Looks like the catalog-operator resource that's a part of OLM (this resource isn't being created during run bundle or run bundle-upgrade) is unable to get the operator condition and is causing the CSV to timeout with the error FATA[0120] Failed to run bundle upgrade: error waiting for CSV to install: timed out waiting for the condition and we have to investigate that.

[J0zi]

@rashmigottipati any update on this? Thank you.

[laxmikantbpandhare]

@J0zi - I am looking into it. I will update you soon.

[jeloba]

I have a similar issue (same error). It started showing up after updating a working setup to OLM 0.20.0 from 0.18.1.

I inspected the cluster and found the following:

• the catalog is installed correctly and running
• the packagemanifest under tests show up
• a subscription is created
• an install plan was created, waiting for manual approval
• approving the install plan is not progressing the installation (probably due to CI job pull credential timeout)

[J0zi]

@laxmikantbpandhare @rashmigottipati any update?

[VenkatRamaraju]

Hey @J0zi,

My suspicion is that the problem is with the version naming. I'm able to install apicurito:v1.0.1 and upgrade it to apicurito:v1.0.2. These operators follow the general semver naming conventions. However, I am not able to upgrade using the examples you provided (timeout). It may be due to the "testing" prefix on the tags of these bundle images.

[vanessarakoczy]

Hi @J0zi,
We are experiencing same issue as well.
Our naming convention is correct, we've tried reverting our OLM back to version 0.18.3 to see if it would resolve anything, but no such luck. Was there any other resolution you were able to find?