KubernetesPodOperator pod_template_file fails with confusing exception (results.metadata is None)

[lgov]

Apache Airflow version:
v1-10-test branch

Kubernetes version (if you are using kubernetes)
v1.19.2

Environment:

• Cloud provider or hardware configuration: Standalone PC
• OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS
• Kernel (e.g. uname -a): GNU/Linux 4.15.0-91-generic x86_64
• Install tools: Docker version 19.03.12
• Others:
  The airflow worker is running in a docker container with an image derived from apache/airflow:latest but the stable apache-airflow package replaced by current v1-10-test branch.

What happened:
We use KubernetesPodOperator in a dag, but when triggering it it always immediately result in an exception.

Folder structure:
dags/basic_pod_task.py
dags/k8s/basic_pod.yaml

Relevant extract of dags/basic_pod_task.py:

with dag:
    t2 = KubernetesPodOperator(
        is_delete_operator_pod=True,
        in_cluster=False,
        cluster_context="kubernetes-admin@kubernetes",
        task_id="task-two",
        get_logs=True,
        pod_template_file="k8s/basic_pod.yaml"
    )

The yaml file is taken from the airflow repo: tests/kubernetes/basic_pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  containers:
    - name: base
      image: perl
      command: ["/bin/bash"]
      args: ["-c", 'echo {\"hello\" : \"world\"} | cat > /airflow/xcom/return.json']
  restartPolicy: Never

Error message:

[2020-09-26 15:25:18,659] {taskinstance.py:901} INFO - Executing <Task(KubernetesPodOperator): task-two> on 2020-09-26T15:24:53.914308+00:00
[2020-09-26 15:25:18,661] {standard_task_runner.py:54} INFO - Started process 423 to run task
[2020-09-26 15:25:18,676] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'basic_test_pod', 'task-two', '2020-09-26T15:24:53.914308+00:00', '--job_id', '93', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/basic_pod_task.py', '--cfg_path', '/tmp/tmpa70_g_m7']
[2020-09-26 15:25:18,677] {standard_task_runner.py:78} INFO - Job 93: Subtask task-two
[2020-09-26 15:25:18,677] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [<function default_action_log at 0x7f651c1be6a8>]
[2020-09-26 15:25:18,683] {settings.py:233} DEBUG - Setting up DB connection pool (PID 423)
[2020-09-26 15:25:18,683] {settings.py:241} DEBUG - settings.configure_orm(): Using NullPool
[2020-09-26 15:25:18,698] {logging_mixin.py:112} INFO - Running <TaskInstance: basic_test_pod.task-two 2020-09-26T15:24:53.914308+00:00 [running]> on host 9b5a0a6250ec
[2020-09-26 15:25:18,711] {__init__.py:101} DEBUG - Preparing lineage inlets and outlets
[2020-09-26 15:25:18,711] {__init__.py:137} DEBUG - inlets: [], outlets: []
[2020-09-26 15:25:18,720] {taskinstance.py:1150} ERROR - 'NoneType' object has no attribute 'name'
Traceback (most recent call last):
  File "/opt/airflow/airflow/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/airflow/airflow/contrib/operators/kubernetes_pod_operator.py", line 280, in execute
    self.pod = self.create_pod_request_obj()
  File "/opt/airflow/airflow/airflow/contrib/operators/kubernetes_pod_operator.py", line 384, in create_pod_request_obj
    pod=self.full_pod_spec,
  File "/opt/airflow/airflow/airflow/kubernetes/pod_generator.py", line 289, in gen_pod
    result.metadata.name = self.make_unique_pod_id(result.metadata.name)
AttributeError: 'NoneType' object has no attribute 'name'
[2020-09-26 15:25:18,721] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=basic_test_pod, task_id=task-two, execution_date=20200926T152453, start_date=20200926T152518, end_date=20200926T152518

What you expected to happen:
I expect that k8s to start up the pod, but we don't get even that far.

I'm guessing that the yaml file is not found, because when I remove the yaml file I get exactly the same error.
The pod starts when I use kubectl apply -f k8s//basic_pod.yaml
Specifying the path as absolute path doesn't solve the problem.

It's not clear what parent path is used for the pod_template_file parameter, but if this is indeed the root cause the exception logging could be more clear.

[mik-laj]

CC: @dimberman

[lgov]

It works when using the absolute path:

pod_template_file="/opt/airflow/dags/basic_pod.yaml"

My mistake was to use the path on the host machine, not the path mapped in the docker container.

I suggest to keep this issue for the confusing error message and lack of clear documentation.

[mik-laj]

We are working on the documentation in another ticket: See: #8970