Resolution error constraints not satisfiable: no operator found from catalog is not cleared after CatalogSource is made available again

[anik120]

Bug Report

What did you do?

1. Run olm in a kind cluster and install an operator:

$ make run-local 

$ kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: my-group
  namespace: default
EOF
operatorgroup.operators.coreos.com/my-group created

$ kubectl apply -f -<<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: camel-k
  namespace: default
spec:
  channel: stable-v2
  name: camel-k
  source: operatorhubio-catalog
  sourceNamespace: olm
  installPlanApproval: Automatic
EOF
subscription.operators.coreos.com/camel-k created

2. Wait for operator to be installed successfully:

$ kubectl get csv 
NAME                      DISPLAY            VERSION   REPLACES                  PHASE
camel-k-operator.v2.2.0   Camel K Operator   2.2.0     camel-k-operator.v2.1.0   Succeeded

$ kubectl get subscription.operators -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,ResolutionFailed:.status.conditions[?(@.type=="ResolutionFailed")].status,MSG:.status.conditions[?(@.type=="ResolutionFailed")].message'
NAMESPACE   NAME      ResolutionFailed   MSG
default     camel-k   <none>             <none>

3. Force a "CatalogSource not found" resolution error:

$  kubectl delete catalogsource operatorhubio-catalog -n olm 
catalogsource.operators.coreos.com "operatorhubio-catalog" deleted
$ k get pods -n olm 
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-86664778f9-pmcxc   1/1     Running   0          7m32s
olm-operator-9d6d59964-gvqt6        1/1     Running   0          7m32s
packageserver-7fdd5ccd9f-rbsrz      1/1     Running   0          7m31s

$ kubectl delete pod catalog-operator-86664778f9-pmcxc 
pod "catalog-operator-86664778f9-pmcxc" deleted

$ kubectl get subscription.operators -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,ResolutionFailed:.status.conditions[?(@.type=="ResolutionFailed")].status,MSG:.status.conditions[?(@.type=="ResolutionFailed")].message' 
NAMESPACE   NAME      ResolutionFailed   MSG
default     camel-k   True               constraints not satisfiable: no operators found from catalog operatorhubio-catalog in namespace olm referenced by subscription camel-k, subscription camel-k exists

4. Recreate the CatalogSource

$ kubectl apply -f deploy/upstream/manifests/latest/0000_50_olm_17-upstream-operators.catalogsource.yaml 
catalogsource.operators.coreos.com/operatorhubio-catalog created

What did you expect to see?
After deleting the new catalog-operator pod to force another resolution, the error message on the subscription to go away

$ kubectl delete pod catalog-operator-86664778f9-mmmqg -n olm 
pod "catalog-operator-86664778f9-mmmqg" deleted

What did you see instead? Under which circumstances?
Error message still exists

$ kubectl get subscription.operators -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,ResolutionFailed:.status.conditions[?(@.type=="ResolutionFailed")].status,MSG:.status.conditions[?(@.type=="ResolutionFailed")].message' 
NAMESPACE   NAME      ResolutionFailed   MSG
default     camel-k   True               constraints not satisfiable: no operators found from catalog operatorhubio-catalog in namespace olm referenced by subscription camel-k, subscription camel-k exists

Environment

• operator-lifecycle-manager version:

master

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-30T06:36:32Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"}

• Kubernetes cluster kind:

kind v0.18.0 go1.20.2 darwin/arm64

Possible Solution

Additional context
Add any other context about the problem here.

[anik120]

Confirmed that OLM v0.24.0 (and v0.23.1) release does not have this issue. When the catalog is deleted, the resolution error shows up on the subscription. When the catalog is re-created, the resolution error is removed

v0.24.0 catalog-operator logs: https://gist.github.com/anik120/9245135fba0e9e7db2ae011257626a94
v0.23.1 catalog-operator logs: https://gist.github.com/anik120/17444d0a9377131d6acf85a6cbebb1d1

It is consistently reproducible from v0.25.0 onwards. v0.25.0 catalog-operator logs: https://gist.github.com/anik120/2dd969a07ce31b0ca7e8f31718b7ebff