New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add flyway variables support via placeholder templater #4026
Changes from all commits
5296ad7
8188f5b
03ca132
1395d95
7e82da5
d200316
b78278b
c7c6932
1c69cda
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -289,6 +289,22 @@ def test__templater_raw(): | |
start_date="'2021-10-01'", | ||
), | ||
), | ||
( | ||
"USE ${flyway:database}.test_schema;", | ||
"flyway_var", | ||
"USE test_db.test_schema;", | ||
{ | ||
"flyway:database": "test_db", | ||
}, | ||
), | ||
( | ||
"SELECT metadata$filename, $1 FROM @stg_data_export_${env_name};", | ||
"flyway_var", | ||
"SELECT metadata$filename, $1 FROM @stg_data_export_staging;", | ||
{ | ||
"env_name": "staging", | ||
}, | ||
), | ||
], | ||
ids=[ | ||
"no_changes", | ||
|
@@ -306,6 +322,8 @@ def test__templater_raw(): | |
"numeric_dollar_with_braces_and_string", | ||
"percent", | ||
"ampersand", | ||
"flyway_var", | ||
"flyway_var", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why does There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I saw some warning to the effect that I can remove and retest if above can be ignored |
||
], | ||
) | ||
def test__templater_param_style(instr, expected_outstr, param_style, values): | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
[sqlfluff] | ||
testing_val=foobar | ||
testing_int=4 | ||
|
||
[sqlfluff:bar] | ||
foo=barbar | ||
|
||
[sqlfluff:templater:placeholder] | ||
param_style = flyway_var | ||
flyway:database = test_db |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
USE ${flyway:database}.test_schema; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see any mention of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thx, will take a look There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. updated with a test, pls take a look There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Curious, could u let me know if this fixture is needed - not sure where/how it'd be used There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's currently not used because the placeholder templater doesn't have a test that reads from file fixtures. See this Can you look at adding a similar test for the placeholder templater? This would test the config delimiter change you made in a realistic way. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. BTW, sorry it took me a while to respond. I had actually written the comment above several days ago, but I forgot to submit the pending comments. Occasionally GitHub confuses me. 😢 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the pointers, I'll have take a look soon There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added a test to config_test.py that should hopefully cover the delimiters codecov |
||
|
||
CREATE OR REPLACE STAGE stg_data_export_${env_name} | ||
URL = 's3://${s3_data_lake_bucket}/${env_name}/exports/stg_data_export' | ||
STORAGE_INTEGRATION = s3_integ_main; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was this added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per the configparser doc, "The first occurrence of a delimiting substring on a line is considered a delimiter" and it assumes both ':' and '=' to be delimiters by default. This would mean that the defining a key called "flyway:database = test_db_name" would split key="flyway" and val="database=test_db_name" :(
I see that all our keys use '=' as delimiter and ':' is only used in section names hence made this change. But, If there's a better way to not impose this globally, pls do let me know
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears the current tests don't test for this change. See my comment on
test/fixtures/templater/placeholder_flyway_var/placeholder_flyway_var_a.sql
for a suggestion on how to test it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mentioned adding another test. Will any of your tests fail if this
delimiters
change is removed? I think it's important to have a test for this.