[Templating]: attributes not defined in `templated_fields` also get value rendered

[charlesdong1991]

Apache Airflow version

2.2.2

What happened

Use a custom operator inherited from BaseOperator, and try to assign a list of sql files to an attribute that is not defined in template_fields, but it still gets rendered, unless the value assigned to the attribute is a string.

Below is the minimum reproducible example and its output, except with a query folder next to dag file, and in this example,

• I expect all attributes starting with suffix of original shouldn’t be rendered
• In fact, only self.original_inputs_str doesn’t get rendered (a string is assigned to it), but the other two, which are assigned a list, somehow both get rendered.

Not sure if I am misunderstanding the templated_fields, or this might be a bug.

Version: 2.2.2
Python: 3.7

What you think should happen instead

Might be a bug? Or I misunderstood how to use it.

How to reproduce

from typing import Sequence

from airflow import DAG
from airflow.models import BaseOperator
from pendulum import datetime


class TestOperator(BaseOperator):
  template_fields: Sequence[str] = ("inputs", "inputs_one", "inputs_two")
  template_ext: Sequence[str] = (".sql",)

  def __init__(self, inputs, inputs_one, inputs_two, **base_operator_kwargs):
    super().__init__(**base_operator_kwargs)

    # should render
    self.inputs = inputs
    self.inputs_one = inputs_one
    self.inputs_two = inputs_two

    # shouldn't render
    self.original_inputs_str = inputs
    self.original_inputs_one = inputs_one
    self.original_inputs_two = inputs_two

  def execute(self, context):
    # render correctly
    print(f"input: {self.inputs}")
    print(f"input 1: {self.inputs_one}")
    print(f"input 2: {self.inputs_two}")

    # expected to be NOT rendered and doesn't get rendered
    print(f"original input str: {self.original_inputs_str}")

    # both expected to be NOT rendered but get rendered
    print(f"original input 1: {self.original_inputs_one}")
    print(f"original input 2: {self.original_inputs_two}")


with DAG(
  dag_id="test",
  description="testing template fields and ext",
  start_date=datetime(2022, 7, 28),
  catchup=False,
  default_args=dict(owner="test"),
  tags=["test"],
) as dag:

  inputs_str = "query/hahaha.sql"
  inputs_one = ["query/hahaha.sql"]
  inputs_two = ["query/haha.sql", "query/hahaha.sql"]

  TestOperator(
    dag=dag,
    task_id="test",
    inputs=inputs_str,
    inputs_one=inputs_one,
    inputs_two=inputs_two,
  )

Operating System

Amazon Linux AMI

Versions of Apache Airflow Providers

No response

Deployment

MWAA

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

I think the root of the problem is that Python strings is immutable, but list is mutable.

from copy import copy

class TestOperator(BaseOperator):
  template_fields: Sequence[str] = ("inputs", "inputs_one", "inputs_two")
  template_ext: Sequence[str] = (".sql",)

  def __init__(self, inputs, inputs_one, inputs_two, **base_operator_kwargs):
    super().__init__(**base_operator_kwargs)

    # should render, create shallow copy of original objects
    # might required deepcopy, if nested object also mutable.
    # from copy import deepcopy
    self.inputs = copy(inputs)
    self.inputs_one = copy(inputs_one)
    self.inputs_two = copy(inputs_two)

    # shouldn't render
    self.original_inputs_str = inputs
    self.original_inputs_one = inputs_one
    self.original_inputs_two = inputs_two

[uranusjr]

Yes, your original_inputs_one and inputs_one (same as two) ultimately point to the same value, and it’s just how Python works. Airflow can’t blindly do copy.copy either since while it solves the problem for built-in types, you’d still hit the same issue if you pass in custom class objects. So I’m inclined to close this as won’t fix.

[charlesdong1991]

ah, I see where it went wrong!!

Great thanks for the answer!!! Appreciate it a lot!! @Taragolis @uranusjr 🙇

[Taragolis]

@charlesdong1991 Just a silly question, why you need keep original values of rendered attributes?

Due to my experience last time when someone want to keep both value did it only for logging purpose, however original values before rendering shown in UI: Task Instances -> Task Instance Detail -> Task Attributes

Might be it also your case?

[charlesdong1991]

Hey, @Taragolis it isn't silly at all, probably I am doing something very silly, haha 😅

So the use case at my company is, when we get the parsed sql query from for instance the file haha.sql, we also want to get this filename used for other logics in our codebase. However, when setting the template_fields and template_ext, we cannot get that original filename anymore, and then we take a work it around to try to assign to a different attribute.

So, if I understand correctly, for this case, we could use ti.xcom_pull for instance, to also let us get the original name for the attribute in template_fields?

Not sure if i look at the right place, but from UI Task Attribute, it seems that i still see the rendered result, instead of original one?

Thanks a lot for your help!! 🙇 And happy weekend!! 🤗