Update query for getting bigquery table schema #661
Conversation
utkarsharma2
requested review from
dimberman,
tatiana,
sunank200,
pankajastro,
pankajkoti and
feluelle
as code owners
utkarsharma2
changed the title
utkarsharma2
changed the title
tests/databases/test_bigquery.py
Outdated
| statement = f"SELECT * FROM {table.metadata.schema}.INFORMATION_SCHEMA.COLUMNS WHERE table_name='{table.name}'" | ||
| # Looking for specific columns in INFORMATION_SCHEMA.COLUMNS as Bigquery can add/remove columns in the table. | ||
| statement = ( | ||
| f"SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM {table.metadata.schema}" |
There was a problem hiding this comment.
What are those extra columns btw? This looks good to be but it could be replaced with the values on this assert: https://github.com/astronomer/astro-sdk/runs/7830499547?check_suite_focus=true#step:11:765
Also I think f-string change has to be done for deepsource
There was a problem hiding this comment.
@sunank200 column_default is the col added. Which is not listed in https://cloud.google.com/bigquery/docs/information-schema-columns
Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
Codecov Report
@@ Coverage Diff @@
## main #661 +/- ##
==========================================
- Coverage 93.29% 92.76% -0.53%
==========================================
Files 41 42 +1
Lines 1670 1701 +31
Branches 210 216 +6
==========================================
+ Hits 1558 1578 +20
- Misses 89 97 +8
- Partials 23 26 +3
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
The Deepsource:Python analysis is suggesting that Line 446 in this file can be converted to f-string. It is not part of your change but can we convert the format clause string to an f-string. https://github.com/astronomer/astro-sdk/pull/661/files#diff-d5242436b93ad662a1ef1f5df4b89b9c3da692fb988142b475fc72c1ddfdf7beR446
Analysis message: https://deepsource.io/gh/astronomer/astro-sdk/run/2044361d-845a-43fc-b418-c1da18b66adb/python/PYL-C0209
|
@kaxil I'm merging this as it will be a blocker for other PRs. Please let me know if you think there should be some changes here. |
|
@pankajkoti I'll raise a separate PR for this, as the changes are not related to the current issue. |
* Updat query for bigquery schema from to specfic cols * Update tests/databases/test_bigquery.py Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>a Description What is the current behavior? There is a failing test case in CI - test_bigquery_create_table_with_columns Can be because of PR - #641 (comment) Action run failure - https://github.com/astronomer/astro-sdk/runs/7830499547?check_suite_focus=true Astro: 1.0.0b1 closes: #660 What is the new behavior? Bigquery added an extra col in the INFORMATION_SCHEMA.COLUMNS and the test broke in CI since we checked for the entire cols involved. Now we are only looking for specific columns, this should make the test more robust. Does this introduce a breaking change? Nope Checklist Created tests that fail without the change (if possible)
* Updat query for bigquery schema from to specfic cols * Update tests/databases/test_bigquery.py Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>a Description What is the current behavior? There is a failing test case in CI - test_bigquery_create_table_with_columns Can be because of PR - #641 (comment) Action run failure - https://github.com/astronomer/astro-sdk/runs/7830499547?check_suite_focus=true Astro: 1.0.0b1 closes: #660 What is the new behavior? Bigquery added an extra col in the INFORMATION_SCHEMA.COLUMNS and the test broke in CI since we checked for the entire cols involved. Now we are only looking for specific columns, this should make the test more robust. Does this introduce a breaking change? Nope Checklist Created tests that fail without the change (if possible)
Description
What is the current behavior?
There is a failing test case in CI - test_bigquery_create_table_with_columns
Can be because of PR - #641 (comment)
Action run failure - https://github.com/astronomer/astro-sdk/runs/7830499547?check_suite_focus=true
Astro: 1.0.0b1
closes: #660
What is the new behavior?
Bigquery added an extra col in the INFORMATION_SCHEMA.COLUMNS and the test broke in CI since we checked for the entire cols involved. Now we are only looking for specific columns, this should make the test more robust.
Does this introduce a breaking change?
Nope
Checklist