Nested templated variables do not always render

[smith-m]

Apache Airflow version:
1.10.14 and 1.10.8.

Environment:
Python 3.6 and Airflow 1.10.14 on sqllite,

What happened:

Nested jinja templates do not consistently render when running tasks. TI run rendering behavior also differs from airflow UI and airflow render cli.

What you expected to happen:

Airflow should render nested jinja templates consistently and completely across each interface. Coming from airflow 1.8.2, this used to be the case.

This regression may have been introduced in 1.10.6 with a refactor of BaseOperator templating functionality.
#5461

Whether or not a nested layer renders seems to differ based on which arg is being templated in an operator and perhaps order. Furthermore, it seems like the render cli and airflow ui each apply TI.render_templates() a second time, creating inconsistency in what nested templates get rendered.

There may be bug in the way BaseOperator.render_template() observes/caches templated fields

How to reproduce it:
From the most basic airflow setup

nested_template_bug.py

from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator

with DAG("nested_template_bug", start_date=datetime(2021, 1, 1)) as dag:

    arg0 = 'level_0_{{task.task_id}}_{{ds}}'
    kwarg1 = 'level_1_{{task.op_args[0]}}'

    def print_fields(arg0, kwarg1):
        print(f'level 0 arg0: {arg0}')
        print(f'level 1 kwarg1: {kwarg1}') 

    nested_render = PythonOperator(
        task_id='nested_render',
        python_callable=print_fields,
        op_args=[arg0, ],
        op_kwargs={
            'kwarg1': kwarg1,
        },
    )

> airflow test c

level 0 arg0: level_0_nested_render_2021-01-01
level 1 kwarg1: level_1_level_0_{{task.task_id}}_{{ds}}

> airflow render nested_template_bug nested_render 2021-01-01

# ----------------------------------------------------------
# property: op_args
# ----------------------------------------------------------
['level_0_nested_render_2021-01-01']

# ----------------------------------------------------------
# property: op_kwargs
# ----------------------------------------------------------
{'kwarg1': 'level_1_level_0_nested_render_2021-01-01'}

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

This seems to be also issue in 2.0:

[2021-01-08 09:46:58,236] {logging_mixin.py:103} INFO - level 0 arg0: level_0_nested_render_2021-01-08
[2021-01-08 09:46:58,239] {logging_mixin.py:103} INFO - level 1 kwarg1: level_1_level_0_{{task.task_id}}_{{ds}}

but it is consistent with rendered values in web ui:

[RosterIn]

I don't know if this is a bug or by design but there are many votes for this in https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros

[turbaszek]

I'm afraid that this is not simple to fix. The main problem I see is that template fields are rendered in the order they are defined in op.template_fields. This means that some values may reference values that were not yet rendered and this will cause a problem.

[smith-m]

One possibility here is using the previous state of each attribute and rendering each templated_field until all fields are stable.
This will not address macro expansion in the stackoverflow example, but would support nested jinja

[turbaszek]

One possibility here is using the previous state of each attribute and rendering each templated_field until all fields are stable.

I'm wondering how this may impact performance. Maybe it would be worth to have proof of concept?

WDYT @kaxil @XD-DENG ?

[XD-DENG]

I agree with @turbaszek on the potential complexity of "fixing" this.

What I'm thinking is: if we can have different method (other than templating) to achieve the same functionality/purpose, we don't have to bother with it. Correct me if I'm wrong please.

[utkarshgupta137]

I was having a similar problem: I wanted to pass a templated argument to a function, but it kept getting passed as raw template instead of rendered one.

The solution is to call the macro inside a pythonoperator & pass the data using xcoms.

def get_date(**kwargs):
    # {{ dag_run.conf.get("date", "{{ ds_no_dash }}") }} doesn't work

    ds = kwargs["ds_no_dash"]
    date = kwargs["dag_run"].conf.get("date", ds)
    kwargs["ti"].xcom_push("date", base_data)

[uranusjr]

If we support nested rendering we’d also need to worry about circular reference, which would require a significantly more elaborated algorithm to properly handle. Dependency resolution is hell, don’t go there if at all possible.

[potiuk]

I looked at this and the problem turned out to be a bit different.

True - we should not even attempt to solve the recursive rendering of fields cross-referencing each other. But this was not the case. This was a kwargs field which referred opargs field - and kwargs was AFTER opargs in the list of templated fields, so it should work in principle, without resolving the recursive problem.

I looked at it and it turned out, this was a problem introduced by #8805 - where instead of the original task we started to use a copy of the task locked for execution. Unfortunately - the task in context was still the original one, not the copy, which caused the rendering problem (as rendered fields in the 'context' task were not updated at all.

This means that if someone would use the context['task'] later in Python Callable or custom operator, the fields were not rendered there either 😱 .

I fixed that in the #18516 by replacing the task in context with the same locked-down copy - not sure if this might have other side effects (but I think it could only be positive side effects :D ).

[potiuk]

With the change from #18516, I got this:

[rkranjancs1014]

"{{var.value.get("const_"+dag_run.conf.get("arg1", ""))}}"

this works for me. here I am getting arg1 one as argument in dag run and creating variable at run time then accessing variable from var.