Render sql code with parameters in BaseSQLDecoratedOperator

[feluelle]

Description

What is the current behavior?

Currently, when using operators inheriting from BaseSQLDecoratedOperator we are unable to view the rendered SQL in the UI.

closes: #1003
depends on: #1004

What is the new behavior?

• add sql and parameters to BaseSQLDecoratedOperator template_fields
• add .sql to BaseSQLDecoratedOperator template_ext
• add check for template_fields and template_ext

Does this introduce a breaking change?

No.

Checklist

• Created tests which fail without the change (if possible)
• Extended the README / documentation, if necessary

[tatiana]

@feluelle I think it could be confusing to end-users to have three methods for creating a table from a SQL statement:

• transform
• transform_file
• transform_sql

I can see the need for having a decorator (transform) x a task, and I understand you didn't want to introduce any breaking changes, but it is confusing overall. What do you think of:

1, Adding a deprecation warning in transform_file
2. Name the new task transform_task (or something else) - so it is clear the difference in comparison to transform - and it could be used both to load from a file and a string?
3. Should we have some tests?

[feluelle]

1, Adding a deprecation warning in transform_file
2. Name the new task transform_task (or something else) - so it is clear the difference in comparison to transform - and it could be used both to load from a file and a string?
3. Should we have some tests?

Yes, actually this makes more sense.

[codecov]

Codecov Report

Base: 92.57% // Head: 93.50% // Increases project coverage by +0.93% 🎉

Coverage data is based on head (03a1719) compared to base (49c08b9).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #897      +/-   ##
==========================================
+ Coverage   92.57%   93.50%   +0.93%     
==========================================
  Files          91       91              
  Lines        4810     4837      +27     
  Branches      476      480       +4     
==========================================
+ Hits         4453     4523      +70     
+ Misses        260      225      -35     
+ Partials       97       89       -8     

Impacted Files	Coverage Δ
...thon-sdk/src/astro/sql/operators/base_decorator.py	95.65% <100.00%> (+0.87%)	⬆️
python-sdk/src/astro/sql/operators/load_file.py	95.14% <0.00%> (-1.95%)	⬇️
python-sdk/src/astro/databases/base.py	92.51% <0.00%> (+4.84%)	⬆️
python-sdk/src/astro/files/locations/amazon/s3.py	92.72% <0.00%> (+5.45%)	⬆️
python-sdk/src/astro/utils/table.py	100.00% <0.00%> (+5.71%)	⬆️
python-sdk/src/astro/databases/snowflake.py	92.24% <0.00%> (+8.33%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[tatiana]

I've worked on this PR independently and with @feluelle, and the tests are now passing.

I also validated that both aql. transform and aql.transform_file are rendering the SQL query in the UI:

Example of aql.transform_file:

Example of aql.transform:

Since it was long approved, I'm merging this PR as soon as the CI checks pass - before it completes its five months anniversary. We can continuously improve and iterate over it with time.

This PR has been long standing and was approved a while ago, we can iterate and improve the implementation over time.