Improve Airflow logging for operator Jinja template processing

[barrywhart]

Description

When an operator uses Jinja templating, debugging issues is difficult because the Airflow task log only displays a stack trace.

Use case/motivation

When there's a templating issue, I'd like to have some specific, actionable info to help understand the problem. At minimum:

• Which operator or task had the issue?
• Which field had the issue?
• What was the Jinja template?

Possibly also the Jinja context, although that can be very verbose.

I have prototyped this in my local Airflow dev environment, and I propose something like the following. (Note the logging commands, which are not present in the Airflow repo.)

Please let me know if this sounds reasonable, and I will be happy to create a PR.

def _do_render_template_fields(
    self,
    parent,
    template_fields,
    context,
    jinja_env,
    seen_oids,
) -> None:
    """Copied from Airflow 2.2.5 with added logging."""
    logger.info(f"BaseOperator._do_render_template_fields(): Task {self.task_id}")
    for attr_name in template_fields:
        content = getattr(parent, attr_name)
        if content:
            logger.info(f"Rendering template for '{attr_name}' field: {content!r}")
            rendered_content = self.render_template(content, context, jinja_env, seen_oids)
+            setattr(parent, attr_name, rendered_content)

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

By all means if you can improve it - feel free. This is a huge pain sometimes !

[potiuk]

Best to iterate on a PR - then you can see the code /tests everything else and comment directly there. So feel free to open a PR about it.

[potiuk]

There is one comment I want to make though - it should be rather logged "after" the fact and only when there are problems. This will complicate the PR a bit but should be relatively easy to implement. We certainly do not want to flood the logs with thousands of rendering logs.

[eladkal]

Isn't this information already available in Rendered tab?

[potiuk]

Some people prefer CLI and logs to see problems I believe :)

[eladkal]

Some people prefer CLI and logs to see problems I believe :)

For sure but Jinja may also contain connection template (secrets). If we decide to add logging then we need to consider how to handle it.

[potiuk]

Logs are redacted - this should be no problem.

[barrywhart]

Thanks, I'll try and create a PR some time in the next 1-2 weeks.

[barrywhart]

PR created!

#25452