control_plane_ee_image still uses "latest" tag in release

[bartowl]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

While image_version is based on DEFAULT_AWX_VERSION environment variable that gets injected during staging process, I have found out, that control_plane_ee_image via _control_plane_ee_image is still hardcoded to awx-ee:latest.
This can cause unexpected behaviour, when used with default EE (which is also awx-ee:latest), as the default EE seems to be pulled with each job run when there is an update to that tag.
After such unaware change of awx-ee:latest image, when the awx-task pod terminates for whatever reason, it will start with never code as intended.

As now finally awx-ee images are tagged with tags matching awx, I would recommend to use the same tag on ee as well.

For me, as a workaround, i use following:

spec:
...
  init_container_image: "{{ _image }}-ee"
  init_container_image_version: "{{ _image_version }}"
  control_plane_ee_image: "{{ _image }}-ee:{{ _image_version }}"
...

but this should not be needed if implemented properly.

AWX Operator version

2.3.0

AWX version

22.4.0

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

k3d 5.4.3, kubernetes version v1.23.6+k3s1

Modifications

no

Steps to reproduce

• install default k3d
• wait for awx-ee:latest tag to move
• run any job, that will fetch updated awx-ee:latest
• delete awx-task pod
• new awx-task will have changed controlplane image

Expected results

After installation of AWX via awx-operator, no image should change automatically until deployment of new awx-operator version

Actual results

awx-ee and control_plane images might get changed when latest tag moves

Additional information

No response

Operator Logs

No response

[fosterseth]

@bartowl I agree, I think the default behavior should be for the control_plane_ee_image to use the _image_version instead of just "latest"

would you mind putting up a PR to address this?

[bartowl]

Hi. Sorry for such delay, but I started preparing a PR and stumbled upon a potential problem.
You see - during build process, you discover the latest AWX release. This of course matches the tag on AWX Image.
But - can we be sure, that together with awx, also the awx-ee image will be available with the same tag?
I mean tagging of awx-ee is rather fresh thing, as through last years there was no tags and everything was only available as latest...
I do not want to cause a problem, when let's say new AWX is released, but that same tag is missing on awx-ee image... I do not know AWX project that well. If this is no issue, then I would not only adjust the tags for control_plane image, but also define the default EE as not (latest) but matching AWX release.

[kurokobo]

F.Y.I., tagging awx-ee is automated as a part of release process of AWX itself. So for the recent releases, awx-ee that have same tag as awx is always on quay.io

• In stage job (triggered by AWX team before release process)
  • Pulling quay.io/ansible/awx-ee:latest, re-tag it as ghcr.io/ansible/awx-ee:<version>, and push it to ghcr.io
  • https://github.com/ansible/awx/blob/fc1b74aa68341f96bb256582d7ced5e9109f6d52/.github/workflows/stage.yml#L96-L100
• In promote job (triggered by creating new release by AWX team)
  • Pulling ghcr.io/ansible/awx-ee:<version>, re-tag it as quay.io/ansible/awx-ee:<version>, and push it to quay.io.
  • https://github.com/ansible/awx/blob/fc1b74aa68341f96bb256582d7ced5e9109f6d52/.github/workflows/promote.yml#L86-L88

As @fosterseth mentioned above, there is a variable _image_version that contains versioned tag (refer to this line, the DEFAULT_AWX_VERSION is passed from stage job above during building AWX Operator), so for your PR simply replacing latest in the default value for _control_plane_ee_image with that variable is good starting point, I guess.

[fosterseth]

@bartowl what @kurokobo pointed out is correct ^ for every awx version tag there should be a corresponding awx-ee version tag

[bartowl]

Great, then i'll prepare a PR shortly, maybe i make it even today...

[Denney-tech]

It would be nice if it was more like the other *_image specs as well:

control_plane_ee_image: quay.io/ansible/awx-ee
control_plane_ee_image_version: 23.8.1

[bartowl]

just as a side note for air-gapped environments - it is not enough to replace all documented images (image, redis_image, ee_images[] etc), as there is also the undocumented (yet) init_container_image and init_container_image_version. Its default is based on _control_plane_ee_image and not control_plane_ee_image so just keep that in mind, otherwise your containers will not be able to start. Maybe it would be worth to mention in the documentation?