Issue with loading files using LoadFileOperator with native_support=True #890
Assignees
Labels
bug
Something isn't working
priority/critical
Critical priority
product/python-sdk
Label describing products
Milestone
Comments
kaxil
added
bug
|
@fritz-astronomer I'm not able to reproduce this on my local, can you share the details like the GCS files and Snowflake Table details OR we can get on a call to try this out? |
|
@utkarsharma2 following up in slack |
|
@utkarsharma2 could you please once check the status on this issue? |
|
Need to try on the env @fritz-astronomer has shared with me. |
2 tasks
utkarsharma2
added a commit
that referenced
this issue
Nov 4, 2022
# Description ## What is the current behavior? Currently, we were not bubbling up the error raised by copying into command in snowflake native path GCS -> Snowflake. closes: #890 ## What is the new behavior? Now we pass a handler that checks for the status of the query returned and if any is failing raise a `DatabaseCustomError` error. ## Does this introduce a breaking change? Nope ### Checklist - [ ] Created tests which fail without the change (if possible) - [ ] Extended the README / documentation, if necessary
|
Thank you!!!! |
utkarsharma2
added a commit
that referenced
this issue
Nov 4, 2022
# Description ## What is the current behavior? Currently, we were not bubbling up the error raised by copying into command in snowflake native path GCS -> Snowflake. closes: #890 ## What is the new behavior? Now we pass a handler that checks for the status of the query returned and if any is failing raise a `DatabaseCustomError` error. ## Does this introduce a breaking change? Nope ### Checklist - [ ] Created tests which fail without the change (if possible) - [ ] Extended the README / documentation, if necessary
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Check why the
'use_native_support'=Trueisn't loading all the files with the LoadFileOperatorRelevant thread https://astronomer.slack.com/archives/C02B8SPT93K/p1663783869151809
I'm seeing an issue in my job with the LoadFileOperator
The file was a GCS Parquet file - I physically checked and there is data in it
The table was a Snowflake TempTable - I checked and it's empty
It then gets merged with a MergeOperator, but given that the temp table was empty, nothing was added
No errors in the logs as far as I can tell.
I have use_native_support=True - should I turn it off and see if that helps?
This happened only occasionally - I loaded ~80 files each for 4 different tables (as fast as airflow could load them), and only some loads were affected
It does appear to happen consistently with the same records missing for at least one case if I rerun the tasks though
Version
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: