KubernetesPodOperator raises exception when pod_template_file contains multiple objects

[everglory99]

In Airflow 2.0.1, we are trying to use pod_template_file in KubernetesPodOperator. The pod_template_file contains several sections specifying ConfigMap, Service and Deployment etc.. However, I am seeing this error from Airflow

raise ComposerError("expected a single document in the stream",
yaml.composer.ComposerError: expected a single document in the stream
but found another document

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[dimberman]

@everglory99 can you please post the YAML file you're using?

[everglory99]

The YAML file is generated by a helm chart our team developed by running helm template. The YAML file looks like

---
apiVersion: v1
kind: ConfigMap 
metadata:
  name: dev-configmap

data:
  SOME_KEY: "SOME_VALUE"
---
apiVersion: v1
kind: Service
metadata:
  name: dev-engine
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
  selector:
    app.kubernetes.io/name: engine
    app.kubernetes.io/instance: dev
---
SOME MORE OBJECTS...

[dimberman]

@everglory99 I don't think you can do that with pod_template file. we expect just a single pod definition. You're trying to launch a pod with an attached service/configmap?

[everglory99]

Yes, that is exactly the use case here. Do you have any suggestions how to achieve this in Airflow? Thanks!

[dimberman]

Hmm so one option would be to create a subclass of the KubernetesPodOperator that takes in file paths for these extra objects and launches them before launching the pod with a pod_template_file that launches JUST the pod and then deletes those objects in a "finally" block to ensure they are gone after the task ends. Would that work?

cc: @kaxil

[kaxil]

Hmm so one option would be to create a subclass of the KubernetesPodOperator that takes in file paths for these extra objects and launches them before launching the pod with a pod_template_file that launches JUST the pod and then deletes those objects in a "finally" block to ensure they are gone after the task ends. Would that work?

cc: @kaxil

@dimberman Would you consider this a feature request then instead of a bug?

[everglory99]

Thanks @dimberman for the suggestions! I am wondering how to launch non-pod objects using Airflow? Should I just use K8s Python client?

[dimberman]

@everglory99 Yep! That should totally work. You can even build a custom operator/donate it if you think it will be useful :)

@kaxil yes I think this is a feature request.

@everglory99 would you be intereseted in building that? I can help get you setup with a dev environment

[jedcunningham]

@everglory99, have you considered exposing a way to add these extra objects into your helm chart, say via extraConfigMaps etc or more generally extraObjects? It seems much saner to update them with helm instead of trying to deploy them with every task run (which is a whole different can of worms, unless you run with parallelism set to 1 😀).

[dimberman]

@jedcunningham good point. Deploying k8s objects in parallel can potentially lead to weird states.

[dimberman]

@everglory99 I'm going to close this but please feel free to reopen if you have more questions