imagePullPolicy: IfNotPresent may lead to image verification errors

[jchaves]

Describe the bug
Current imagePullPolicy for all components: "ifNotPresent"

We believe that the current configuration can lead to inconsistent results (observed already on passes to prod): when it comes to verifying the image of the running container, we have seen it already failing in a couple of instances:

Image verification

To Reproduce
Steps to reproduce the behavior:
It does not happen everytime, but the steps would roughly be:

1. Prepare to deploy to prod
2. Observe that the imagestreams get imported
3. Rollout reported as succesfull
4. Image verification fails (reporting that the running container is using an old (and sometimes no longer existing in the namespace's Imagestream list) ImageID

We believe this might be caused by some nodes not correctly pulling the image, maybe considering it already present.

Expected behavior
Images should be always pulled to ensure integrity

Screenshots

Affected version (please complete the following information):

• OpenShift: 4.10
• OpenDevStack 4.0

Log Output
When the error occurs, the log will output something like

Container '${containerName}' is using image '${runningImageSha}' which differs from defined image '${definedImageSha}'.

To summarize: we think it would be beneficial to switch from "imagePullPolicy: IfNotPresent" to "imagePullPolicy: always" to avoid these inconsistencies, even when this could potentially impact performance.