New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support loading metadata columns from stage into table for Snowflake #2023
Support loading metadata columns from stage into table for Snowflake #2023
Conversation
The transformed COPY INTO command executed is like the below when metadata_columns are specified:
|
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #2023 +/- ##
==========================================
+ Coverage 89.54% 90.28% +0.73%
==========================================
Files 75 75
Lines 4296 4324 +28
Branches 531 537 +6
==========================================
+ Hits 3847 3904 +57
+ Misses 354 332 -22
+ Partials 95 88 -7
Flags with carried forward coverage won't be shown. Click here to find out more.
☔ View full report in Codecov by Sentry. |
34d9023
to
9f0810c
Compare
Adds support to load metadata columns like METADATA$FILENAME, METADATA$FILE_ROW_NUMBER, etc. from stage into target table while natively loading files into Snowflake tables. Read more at: https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table Note that, you cannot specify both `validation_mode` and `metadata_columns` together in the Snowflake load options because when we need to load `metadata_columns`, we need to explicitly name the metadata columns in the `COPY INTO` sql statement and such a transformed SQL statement does not allow specifying `VALIDATION_MODE` with it. It's a limitation for Snowflake queries. The transformed SQL appears like in the snippet in the following link: https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table closes: #1982
Co-authored-by: Utkarsh Sharma <utkarsharma2@gmail.com>
…ribute errors for handlers
000adc5
to
ec9080e
Compare
Nice! Thank you! |
Adds support to load metadata columns like METADATA$FILENAME,
METADATA$FILE_ROW_NUMBER, etc. from stage into target table
while natively loading files into Snowflake tables. Read more at:
https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table
Note that, you cannot specify both
validation_mode
andmetadata_columns
together in the Snowflake load optionsbecause when we need to load
metadata_columns
, we needto explicitly name the metadata columns in the
COPY INTO
sql statement and such a transformed SQL statement does not
allow specifying
VALIDATION_MODE
with it. It's a limitationfor Snowflake queries.
The transformed SQL appears like in the snippet in the following link:
https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table
closes: #1982