datastax/cass-config-builder pull error on OCP

[mgoerens]

What happened?

When deploying the certified DataStax Kubernetes Operator for Apache Cassandra v1.14.0 on OCP 4.12.5, following issue has been reported:

Using default value for the CassandraDatacenter CR, the initContainer doesn't come up due to the container image not being found, with the error message Back-off pulling image "datastax/cass-config-builder:1.0-ubi7"

$ oc get po -n openshift-operators
development-dc1-rack1-sts-0                         0/2
Init:ImagePullBackOff   0          19m
development-dc1-rack2-sts-0                         0/2
Init:ImagePullBackOff   0          19m
development-dc1-rack3-sts-0                         0/2
Init:ImagePullBackOff   0          19m

After some investigation, we found that the 1.0-ubi7 tag is not present on the registry.connect.redhat.com repo:

 $ podman search --list-tags registry.connect.redhat.com/datastax/cass-config-builder
NAME                                                                                               TAG
registry.connect.redhat.com/datastax/cass-config-builder  1.0.0
registry.connect.redhat.com/datastax/cass-config-builder  latest
registry.connect.redhat.com/datastax/cass-config-builder  1.0.0-2
registry.connect.redhat.com/datastax/cass-config-builder  1.0.3-ubi7-2

(It is however present on docker.io)

Could you update the registry.connect.redhat.com/datastax/cass-config-builder accordingly and push the 1.0-ubi7 tag there ?

What did you expect to happen?

No response

How can we reproduce it (as minimally and precisely as possible)?

See above

cass-operator version

v1.14.0

Kubernetes version

OCP 4.12.5

Method of installation

Operator Hub Certified Catalog

Anything else we need to know?

No response

[burmanm]

I'll add it on the 1.16 release to automatically pull from registry (with the hash code).

[mgoerens]

@burmanm My understanding is that the datastax/cass-config-builder:1.0-ubi7 image is missing from redhat.registry.com and needs to be pushed there.

[burmanm]

You're right, but currently we push releases from this repo when making one (and do the preflight check which is required to push there).

https://github.com/k8ssandra/cass-operator/blob/master/.github/workflows/release.yml#L121

The config-builder project does not have the ability to automatically push new versions there (or rebuild the existing ones). And as such, we might need to do it from this repo. And we also use the SHA (as recommended/required) in those builds:

https://github.com/k8ssandra/cass-operator/blob/master/scripts/release-certified-bundles.sh#L29

I'll try to see later if I have the preflight installed on any of my machines to check if I could manually push it meanwhile. All those older images were pushed when the process was different.

[mgoerens]

Understood. Please let me know if there's any help I can provide with the preflight tool or around certification in general.

Thanks for looking into this !

[burmanm]

Hey, this should be available now:

https://catalog.redhat.com/software/containers/5ec5a839293738682033f76c?architecture=amd64&tag=1.0-ubi7&container-tabs=overview

[mgoerens]

Thanks @burmanm

The initContainer gets created now.

[chenweisomebody126]

When I ran docker pull, I received error.

 % docker pull datastax/cass-config-builder:1.0-ubi7 
1.0-ubi7: Pulling from datastax/cass-config-builder
no matching manifest for linux/arm64/v8 in the manifest list entries

[burmanm]

There is no ubi7 arm64 image. You can use the 1.0 tag without suffix on arm

[chenweisomebody126]

There is no ubi7 arm64 image. You can use the 1.0 tag without suffix on arm

Thanks for replying. I was following this https://docs.k8ssandra.io/install/local/single-cluster-helm/ , but ran into the errror

% kubectl get pods -n k8ssandra-operator                       

NAME                                                READY   STATUS                  RESTARTS   AGE
demo-dc1-default-sts-0                              0/2     Init:ImagePullBackOff   0          4m24s
demo-dc1-default-sts-1                              0/2     Init:ImagePullBackOff   0          4m24s
demo-dc1-default-sts-2                              0/2     Init:ImagePullBackOff   0          4m24s
k8ssandra-operator-86dc65b869-7zlhx                 1/1     Running                 0          6m8s
k8ssandra-operator-cass-operator-67875bf9fb-9b7rw   1/1     Running                 0          6m8s

And then I double checked the status

  Warning  Failed     2m43s (x4 over 4m14s)  kubelet            Failed to pull image "datastax/cass-config-builder:1.0-ubi7": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/datastax/cass-config-builder:1.0-ubi7": no match for platform in manifest: not found

In this case, how should I resolve it? override the chart k8ssandra https://helm.k8ssandra.io/stable with something else?

[burmanm]

When using arm, you need to override the config-builder image. There's a ConfigMap that's deployed (ends with -manager-config) and there's a part:

  image_config.yaml: |
    apiVersion: config.k8ssandra.io/v1beta1
    kind: ImageConfig
    images:
      system-logger: "k8ssandra/system-logger:v1.16.0"
      config-builder: "datastax/cass-config-builder:1.0-ubi7

Change the config-builder to one without the -ubi7 and restart cass-operator (kubectl rollout restart deployment ..)

[chenweisomebody126]

@burmanm Thanks for answering me. I think there is still a bug k8ssandra/k8ssandra-operator#1084 that makes me fail to spin up the stargate service .