Support for operator running outside of roxie control

[mclasmeier]

e.g., for letting the engineer use make -C operator/ run for quick testing of operator code without waiting for operator images.

Manual testing

I am using this command

kubectl get deployments,daemonsets -n acs-central -o json | jq -r '.items[] |  
"\(.kind)/\(.metadata.name):\n" +
(if .spec.template.spec.initContainers then
(.spec.template.spec.initContainers | map("  \(.name): \(.image)") | join("\n")) + "\n"
else "" end) +
(.spec.template.spec.containers | map("  \(.name): \(.image)") | join("\n")) + "\n"
'

for retrieving image-per-container listings. I am running the operator locally from a dirty tag with no published images corresponding to it.

❯ kc -n acs-central get pods                                   
NAME                                  READY   STATUS    RESTARTS        AGE
central-549f58dc4f-fzxfp              1/1     Running   0               7m25s
central-db-54fbbb4885-6nlsn           1/1     Running   0               7m25s
config-controller-6f6dd5858f-6mjcz    1/1     Running   0               7m24s
scanner-9b76868b6-wcq8q               0/1     Running   0               7m25s
scanner-db-54969494f7-p66rv           1/1     Running   0               7m24s
scanner-v4-db-56bc95548-85djt         1/1     Running   0               7m25s
scanner-v4-indexer-56b984f7cc-jc6tn   1/1     Running   2 (7m13s ago)   7m24s
scanner-v4-matcher-69d4b89f4d-wtmqc   1/1     Running   2 (7m13s ago)   7m25s
❯ kc -n acs-sensor get pods 
NAME                                  READY   STATUS    RESTARTS        AGE
admission-control-c8d778dd7-ccsh7     1/1     Running   0               6m12s
collector-2fpbf                       2/2     Running   0               6m15s
collector-mr687                       2/2     Running   0               6m15s
collector-znk74                       2/2     Running   0               6m15s
scanner-5b4c9bbb8f-zkq4c              1/1     Running   0               6m13s
scanner-db-86d7994f78-5s9pd           1/1     Running   0               6m13s
scanner-v4-db-848c7f844c-w5bmr        1/1     Running   0               6m14s
scanner-v4-indexer-6fd87d5bcb-7gbj7   1/1     Running   2 (5m54s ago)   6m13s
sensor-6795d4cd6c-xxrr5               1/1     Running   0               6m14s

(scanner v2 on the central side is just very slow to come up... I checked the scanner-db logs, it is executing the psql initialization -> correct image.)

For acs-central:

Deployment/central:
  central: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569

Deployment/central-db:
  init-db: quay.io/rhacs-eng/central-db:4.10.x-537-g0405dc7569
  central-db: quay.io/rhacs-eng/central-db:4.10.x-537-g0405dc7569

Deployment/config-controller:
  manager: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569

Deployment/scanner:
  scanner: quay.io/rhacs-eng/scanner:2.38.x-24-gcd5fb7a6d1

Deployment/scanner-db:
  init-db: quay.io/rhacs-eng/scanner-db:2.38.x-24-gcd5fb7a6d1
  db: quay.io/rhacs-eng/scanner-db:2.38.x-24-gcd5fb7a6d1

Deployment/scanner-v4-db:
  init-db: quay.io/rhacs-eng/scanner-v4-db:4.10.x-537-g0405dc7569
  db: quay.io/rhacs-eng/scanner-v4-db:4.10.x-537-g0405dc7569

Deployment/scanner-v4-indexer:
  indexer: quay.io/rhacs-eng/scanner-v4:4.10.x-537-g0405dc7569

Deployment/scanner-v4-matcher:
  matcher: quay.io/rhacs-eng/scanner-v4:4.10.x-537-g0405dc7569

For acs-sensor:

Deployment/admission-control:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  admission-control: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569

Deployment/scanner:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  scanner: quay.io/rhacs-eng/scanner-slim:2.38.x-24-gcd5fb7a6d1

Deployment/scanner-db:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  init-db: quay.io/rhacs-eng/scanner-db-slim:2.38.x-24-gcd5fb7a6d1
  db: quay.io/rhacs-eng/scanner-db-slim:2.38.x-24-gcd5fb7a6d1

Deployment/scanner-v4-db:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  init-db: quay.io/rhacs-eng/scanner-v4-db:4.10.x-537-g0405dc7569
  db: quay.io/rhacs-eng/scanner-v4-db:4.10.x-537-g0405dc7569

Deployment/scanner-v4-indexer:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  indexer: quay.io/rhacs-eng/scanner-v4:4.10.x-537-g0405dc7569

Deployment/sensor:
  crs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  sensor: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569

DaemonSet/collector:
  init-tls-certs: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569
  collector: quay.io/rhacs-eng/collector:3.23.x-52-gd22ed884d2
  compliance: quay.io/rhacs-eng/main:4.10.x-537-g0405dc7569

[porridge]

What do image overlays have to do with deployOperator?

[mclasmeier]

This PR is supposed to allow the following flow:
eem is that this does not produce deployable operands out of the box, because they will reference image with some non-existant tag. Either with -dirty suffix or with some commit ID, which does not have CI-built images already.

So, roxie can do the following: It will simply always specify the image refs using overlays. In the usual case, this will be a no-op, because the image refs in the overlays are exactly what the operator would deploy anyway, in case running operator version == to be deployed version, specified with MAIN_IMAGE_TAG. But in case you are using something like make -C operator/ run, this overlay will actually change the refs in a way so that you can actually deployable pods.

[porridge]

What about scanner v2?
I'm not sure enumerating every container and image is a good idea:

• the list of deployed deployments may differ between invocations depending on the CR settings (?)
• not very sustainable in the long term..

[mclasmeier]

Scanner v2 uses different tags.

[mclasmeier]

What about scanner v2?
I'm not sure enumerating every container and image is a good idea:

• the list of deployed deployments may differ between invocations depending on the CR settings (?)
• not very sustainable in the long term..

I think there is no other way than listing all images which use the main image tag. This is done also in the operator of the "related images" pinning.

[mclasmeier]

Regarding:

the list of deployed deployments may differ between invocations depending on the CR settings (?)

True, that's why I am marking the overlays as optional. They just need to cover all image refs which contain a main image tag.

[porridge]

LGTM modulo two nits inline

[mclasmeier]

@porridge Dunno, if you would like to review again, I can also just continue with merging.

[porridge]

One thing that worries me is that we are coupling roxie to the exact list of containers in ACS... Maybe this is a case for https://issues.redhat.com/browse/ROX-26566

[mclasmeier]

One thing that worries me is that we are coupling roxie to the exact list of containers in ACS... Maybe this is a case for https://issues.redhat.com/browse/ROX-26566

True. I just think that the pros of this PR clearly outweigh the cons (maintaining the container list). It makes it so convenient during development & testing to be able to run the operator locally and still be able to deploy ACS using that operator without any further hassle.

[vladbologa]

I'm not sure enumerating every container and image is a good idea:

• the list of deployed deployments may differ between invocations depending on the CR settings (?)
• not very sustainable in the long term..

Looks like @porridge was right with this concern, and I guess using optional doesn't always help.

I have a PR where I am removing init-tls-certs, and I can't deploy it with roxie. I assume it also wouldn't work with older ACS versions that don't have this init container.