KubernetesPodOperator templating the image parameter

[HarryDunham]

Apache Airflow version: 1.10.11

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.17

Environment: AWS

Background:
Per the KubernetesPodOperator template fields, image is listed as a parameter which can be templated. However, this is not reflected in the parameters documentation since it is missing the typical (templated) specifier within the docs. Therefore, it's ambiguous whether it can be templated or not.

What happened:
A task was defined via the KubernetesPodOperator with a jinja template within the image parameter:

t0 = KubernetesPodOperator(
    task_name='dummy_task',
    task_id='dummy_id',
    name='dummy-pod',
    namespace='default',
    dag=dag,
    affinity=affinity,
    tolerations=tolerations,
    image='ACC_ID.dkr.REGION.amazonaws.com/IMAGE:{{ dag_run.conf.image_version }}'
)

Yet the template was not processed through the jinja templating engine. Per the logs:

Failed to apply default image tag "ACC_ID.dkr.ecr.REGION.amazonaws.com/IMAGE_NAME:{{ dag_run.conf.image_version }}": couldn't parse image reference "ACC_ID.dkr.ecr.REGION.amazonaws.com/IMAGE_NAME:{{ dag_run.conf.image_version }}": invalid reference format

What you expected to happen:
The image string to resolve to ACC_ID.dkr.REGION.amazonaws.com/IMAGE:IMAGE_VERSION.

How to reproduce it:
Create a task via the KubernetesPodOperator and supply a jinja template to the image parameter of the operator.

Conclusion:
Should the template fields be altered to reflect this behavior or is this a bug and not working as intended?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[HarryDunham]

After updating our deployment to Airflow 1.10.14, we've found that this appears to be an issue related to airflow version 1.10.11. For reference, pod events locally running on minikube with airflow version 1.10.11:

Events:
  Type     Reason         Age               From               Message
  ----     ------         ----              ----               -------
  Normal   Scheduled      16s               default-scheduler  Successfully assigned default/dummy-task-pod-0599a925b2bd40fdbf8e3779ce0ea2e9 to minikube
  Warning  InspectFailed  0s (x3 over 15s)  kubelet            Failed to apply default image tag "ACC_ID.dkr.ecr.REGION.amazonaws.com/IMAGE:{{ dag_run.conf.image_verison }}": couldn't parse image reference "ACC_ID.dkr.ecr.REGION.amazonaws.com/IMAGE:{{ dag_run.conf.image_verison }}": invalid reference format
  Warning  Failed         0s (x3 over 15s)  kubelet            Error: InvalidImageName

versus pod events running on 1.10.14:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  27s   default-scheduler  Successfully assigned default/dummy-task-pod-62cdce6aa5654a7da4b5c9abfe8cf2ed to minikube
  Normal  Pulled     26s   kubelet            Container image "ACC_ID.dkr.ecr.REGION.amazonaws.com/IMAGE:IMAGE_VERSION" already present on machine
  Normal  Created    26s   kubelet            Created container base
  Normal  Started    26s   kubelet            Started container base

Therefore, I assume this was fixed within 1.10.12 with the other breaking changes. So, I'm closing this issue.

[HarryDunham]

Per this #10068 change listed under the 1.10.12 changelog.