Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@task decorator does not handle "templates_exts" argument correctly (files are not rendered) #22072

Closed
1 of 2 tasks
ghost opened this issue Mar 8, 2022 · 7 comments · Fixed by #26390
Closed
1 of 2 tasks
Assignees
Labels

Comments

@ghost
Copy link

ghost commented Mar 8, 2022

Apache Airflow version

2.2.2

What happened

When I use Taskflow, and especially the @task decorator to create a task, the template files are not correctly rendered : I still have the filename inside templates_dict instead of the templated file content, even though I supplied a valid templates_exts argument (at least one which works with PythonOperator).

I do not encounter the same issue using the old PythonOperator syntax.

What you expected to happen

I expect that the behavior of the @task decorator would be the same as the PythonOperator class, or at least have a workaround explained in the documentation such as other arguments I would need to provide.

How to reproduce

Sample code below :

from airflow.decorators import task

@task(templates_dict={"query": "sql/sample.sql"}, templates_exts=[".sql"])
def aggregate_logs(**kwargs):
    logging.info("query: %s", str(kwargs["templates_dict"]["query"]))

Which returns INFO - query: sql/sample.sql.

However, if I use a PythonOperator with the old syntax it works :

from airflow.operators.python import PythonOperator

def aggregate_logs(**kwargs):
    logging.info("query: %s", str(kwargs["templates_dict"]["query"]))

aggregate_task = PythonOperator(
    python_callable=aggregate_logs,
    templates_dict={"query": "sql/sample.sql"},
    templates_exts=[".sql"],
)

I get the entire templated file inside kwargs["templates_dict"]["query"], not just its name.

Operating System

AWS MWAA

Versions of Apache Airflow Providers

Not sure if relevant here since I only use Airflow packages in the example.

Deployment

MWAA

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@ghost ghost added area:core kind:bug This is a clearly a bug labels Mar 8, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 8, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

@xingularity
Copy link
Contributor

I am interested in this issue and I would like to work on it.

@esthomw
Copy link
Contributor

esthomw commented Sep 5, 2022

@xingularity do you maybe have some updates for that issue?

@potiuk
Copy link
Member

potiuk commented Sep 11, 2022

If there is no response from @xingularity, maybe you want to take over @esthomw ?

@Bowrna
Copy link
Contributor

Bowrna commented Sep 12, 2022

I could try fixing this issue if @xingularity and @esthomw do not want to take over.

@potiuk
Copy link
Member

potiuk commented Sep 12, 2022

Feel free @Bowrna :) - unless @xingularity raises a hand (and @esthomw - you could also help with reviewing and testing the change if you don't feel like implementing it).

@esthomw
Copy link
Contributor

esthomw commented Sep 13, 2022

@potiuk I would love to address that issue, however, I am not sure if I currently have enough time. In that case please go ahead @Bowrna :) I will try to help with reviewing and testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants