Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapped KubernetesPodOperater not rendering nested templates #25588

Closed
2 tasks done
bdsoha opened this issue Aug 8, 2022 · 3 comments · Fixed by #25599
Closed
2 tasks done

Mapped KubernetesPodOperater not rendering nested templates #25588

bdsoha opened this issue Aug 8, 2022 · 3 comments · Fixed by #25599
Labels
area:core kind:bug This is a clearly a bug

Comments

@bdsoha
Copy link
Contributor

bdsoha commented Aug 8, 2022

Apache Airflow version

2.3.3

What happened

Nested values, such as env_vars for the KubernetesPodOperater are not being rendered when used as a dynamically mapped operator.

Assuming the following:

op = KubernetesPodOperater.partial(
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
    # Other arguments
).expand(arguments=[[1], [2]])

The Rendered Template results for env_vars should be:

("[{'name': 'AWS_ACCESS_KEY_ID', 'value': 'some-super-secret-value', 'value_from': None}]")

Instead the actual Rendered Template results for env_vars are un-rendered:

("[{'name': 'AWS_ACCESS_KEY_ID', 'value': '{{ var.value.aws_access_key_id }}', 'value_from': None}]")

This is probably caused by the fact that MappedOperator is not calling KubernetesPodOperater._render_nested_template_fields.

What you think should happen instead

No response

How to reproduce

No response

Operating System

Ubuntu 18.04

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@bdsoha bdsoha added area:core kind:bug This is a clearly a bug labels Aug 8, 2022
@uranusjr
Copy link
Member

uranusjr commented Aug 8, 2022

MappedOperator inherits AbstractOperator, which calls _render_nested_template_fields via render_template. So the analysis seems off.

@uranusjr
Copy link
Member

uranusjr commented Aug 8, 2022

Does this work?

KubernetesPodOperater(
    task_id=...,
    ...,
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
)

@bdsoha
Copy link
Contributor Author

bdsoha commented Aug 8, 2022

Does this work?

KubernetesPodOperater(
    task_id=...,
    ...,
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
)

Yes, as mentioned above the issue is only when using the operator as a mapped-operator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants