[BUG] Can't test community-operators for OpenShift 4.6

[wallrj]

There was a bug in the community-operators catalog source which got included in OpenShift 4.6.15, the version that was bundled by crc:

• Bug 1937986: wrong community catalog image reference operator-framework/operator-marketplace#384
• https://access.redhat.com/solutions/6470961
• https://bugzilla.redhat.com/show_bug.cgi?id=1937986

This leaves me with no easy way to test the cert-manager community operator on OpenShift 4.6.

Would you consider releasing a new patch version of crc 1.22 with a fixed version of OpenShift 4.6?
Or explain how to upgrade the openshift once it has been started.

General information

• OS: Linux
• Hypervisor: KVM
• Did you run crc setup before starting it: Yes
• Running CRC on: VM

CRC version

[crc@crc-4-6 ~]$ bin/crc-1.22.0 version
CodeReady Containers version: 1.22.0+6faff76f
OpenShift version: 4.6.15 (embedded in executable)

CRC status

[crc@crc-4-6 ~]$ bin/crc-1.22.0 status
CRC VM:          Running
OpenShift:       Degraded (v4.6.15)
Disk Usage:      15.28GB of 32.72GB (Inside the CRC VM)
Cache Usage:     14.31GB
Cache Directory: /home/crc/.crc/cache

CRC config

[crc@crc-4-6 ~]$ bin/crc-1.22.0 config view
- consent-telemetry                     : yes
- cpus                                  : 8
- disable-update-check                  : true
- memory                                : 24576
- pull-secret-file                      : build/pull-secret

Host Operating System

[crc@crc-4-6 ~]$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.4"

Steps to reproduce

1. crc setup
2. crc start
3. Browse to https://console-openshift-console.apps-crc.testing/operatorhub/ns/openshift-operators
4. Search for cert-manager

Expected

I expected to see the cert-manager community-operators entry

Actual

No community operators are visible

Logs

[crc@crc-4-6 ~]$ oc -n openshift-marketplace describe catalogsources community-operators
Name:         community-operators
Namespace:    openshift-marketplace
Labels:       <none>
Annotations:  operatorframework.io/managed-by: marketplace-operator
API Version:  operators.coreos.com/v1alpha1
Kind:         CatalogSource
Metadata:
  Creation Timestamp:  2021-02-02T06:21:00Z
  Generation:          5
  Managed Fields:
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:operatorframework.io/managed-by:
      f:spec:
        .:
        f:displayName:
        f:icon:
          .:
          f:base64data:
          f:mediatype:
        f:image:
        f:priority:
        f:publisher:
        f:sourceType:
        f:updateStrategy:
          .:
          f:registryPoll:
            .:
            f:interval:
    Manager:      marketplace-operator
    Operation:    Update
    Time:         2021-11-04T01:19:39Z
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:connectionState:
          .:
          f:address:
          f:lastConnect:
          f:lastObservedState:
        f:latestImageRegistryPoll:
        f:registryService:
          .:
          f:createdAt:
          f:port:
          f:protocol:
          f:serviceName:
          f:serviceNamespace:
    Manager:         catalog
    Operation:       Update
    Time:            2021-11-04T01:49:32Z
  Resource Version:  60676
  Self Link:         /apis/operators.coreos.com/v1alpha1/namespaces/openshift-marketplace/catalogsources/community-operators
  UID:               78c3fa40-6548-4542-870d-30e21c51786c
Spec:
  Display Name:  Community Operators
  Icon:
    base64data:  
    Mediatype:   
  Image:         registry.redhat.io/redhat/community-operator-index:latest
  Priority:      -400
  Publisher:     Red Hat
  Source Type:   grpc
  Update Strategy:
    Registry Poll:
      Interval:  10m0s
Status:
  Connection State:
    Address:                   community-operators.openshift-marketplace.svc:50051
    Last Connect:              2021-11-04T01:49:32Z
    Last Observed State:       TRANSIENT_FAILURE
  Latest Image Registry Poll:  2021-11-04T00:30:38Z
  Registry Service:
    Created At:         2021-02-02T06:21:02Z
    Port:               50051
    Protocol:           grpc
    Service Name:       community-operators
    Service Namespace:  openshift-marketplace
Events:                 <none>

[crc@crc-4-6 ~]$ oc -n openshift-marketplace describe pod community-operators-xpdnf
...
Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       9h                 default-scheduler  Successfully assigned openshift-marketplace/community-operators-xpdnf to crc-ctj2r-master-0
  Normal   AddedInterface  9h                 multus             Add eth0 [10.217.0.46/23]
  Normal   Pulling         9h (x4 over 9h)    kubelet            Pulling image "registry.redhat.io/redhat/community-operator-index:latest"
  Warning  Failed          9h (x4 over 9h)    kubelet            Failed to pull image "registry.redhat.io/redhat/community-operator-index:latest": rpc error: code = Unknown desc = Error reading manifest latest in registry.redhat.io/redhat/community-operator-index: unknown: Tag latest was deleted or has expired. To pull, revive via time machine
  Warning  Failed          9h (x4 over 9h)    kubelet            Error: ErrImagePull
  Normal   BackOff         8h (x276 over 9h)  kubelet            Back-off pulling image "registry.redhat.io/redhat/community-operator-index:latest"
  Warning  Failed          8h (x341 over 9h)  kubelet            Error: ImagePullBackOff

[cfergeau]

Unfortunately we don't provide support for older OpenShift versions. Your best bet would be to regenerate your own OpenShift 4.6 bundle using https://github.com/code-ready/snc/ , but that can be an involved process.
I assume newer crc versions aren't an option for you?

[wallrj]

Unfortunately we don't provide support for older OpenShift versions. Your best bet would be to regenerate your own OpenShift 4.6 bundle using https://github.com/code-ready/snc/ , but that can be an involved process. I assume newer crc versions aren't an option for you?

Thanks. I just need a way to quickly run OpenShift 4.6 to test that the cert-manager community operator works on that version.
I am under the impression that crc-1.22.0 is the last version that bundled OpenShift 4.6:

• https://mirror.openshift.com/pub/openshift-v4/clients/crc/1.22.0/release-info.json
• https://mirror.openshift.com/pub/openshift-v4/clients/crc/1.23.1/release-info.json

Is there a way to create an OpenShift 4.6 cluster using the latest version of crc?

[cfergeau]

After reading https://bugzilla.redhat.com/show_bug.cgi?id=1937986 , it seems the error you are seeing is because the latest tag was removed from registry.redhat.io/redhat/community-operator-index, but there were private comments yesterday saying they are trying to readd it. So the issue you are seeing might "fix itself" in the coming days ;)

[wallrj]

I've verified that I can now see the community-operators on OpenShift 4.6.15 booted using crc-122.0. Thanks!