Allow users to disable schema check and creation on load_file
#1922
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #1922 +/- ##
==========================================
- Coverage 85.31% 84.81% -0.51%
==========================================
Files 104 104
Lines 5952 5959 +7
Branches 677 678 +1
==========================================
- Hits 5078 5054 -24
- Misses 735 762 +27
- Partials 139 143 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
☔ View full report in Codecov by Sentry. |
tatiana
requested review from
dimberman,
utkarsharma2,
sunank200,
pankajastro and
pankajkoti
as code owners
|
@tatiana Do you think a global setting as a default can be helpful? like an env variable? |
@utkarsharma2 that's a great idea, I'll add to this PR. |
|
@utkarsharma2 I added the global config, please, let me know your thoughts! |
utkarsharma2
reviewed
May 5, 2023
utkarsharma2
approved these changes
May 5, 2023
kaxil
approved these changes
May 5, 2023
tatiana
added a commit
that referenced
this pull request
May 5, 2023
This is a follow-up for #1922. In that PR we allowed users to skip schema check & creation for `aql.load_file`, but we missed the fact that `aql.transform` and `aql.transform_file` had the same issue. This PR aims to address this limitation. Changes included in this PR: * Rename config `load_table_schema_exists` to `assume_schema_exists` * Rename (`load_file`) argument `schema_exists` to `assume_schema_exists` * Refactor where the check for `assume_schema_exists` happens. Before, it happened only inside the `load_file_to_table`. Now, it is part of `create_schema_if_applicable`. This makes this feature available in the `aql.transform` task as well * Rename `Database.create_schema_if_needed` to `Database.create_schema_if_applicable` * Expose `assume_schema_exists` in `aql.transform` * Release 1.7.0a2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support running
load_filewithout checking if the table schema exists or trying to create it.Recently a user reported that the cost of checking if the schema exists is very high for Snowflake:
"I have a (
load_file) task that took 1:36 minutes to run, and it was 1:30 running the information schema query."This is likely happening for other databases as well.
Introduce two ways of disabling schema checks:
On a per-task basis, by exposing the argument
schema_existsinaql.load_fileWhen this argument is
True, the SDK will not check if the schema exists or try to create it.It is
Falseby default, and the Python SDK will behave as of 1.6 (running schema check and, if needed, trying to create the schema)Globally, by exposing the Airflow configuration
load_table_schema_existsin the[astro-sdk]section. This can also be set using the environment variableAIRFLOW__ASTRO_SDK__LOAD_TABLE_SCHEMA_EXISTS. The global configuration can be overridden per task, using [1].Closes: #1921