-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CIVP-24576] FIX relax SQL type checking in civis_file_to_table
#439
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@skomanduri Tagging you for review, since there's a small chance that this PR requires internal Platform knowledge that you'd know better than anyone else
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very comprehensive!!! Just have some minor style suggestions and then one question about the inconsistent-headers logic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jacksonlee-civis Got it. Thanks for explaining your process. I was able to do some verification myself using the same methodology and script you provided and I couldn't get this to break with the cases I tried, so I think this is good to go! I just found one small issue where the column name might not get pulled from the file but might be auto-generated instead. Feel free to code it differently if you want.
...rather than throwing an error for the first problematic file found, and thereby masking potential subsequent, similar errors and slowing down the entire work/debugging process
@skomanduri Thank you for catching the column name issue -- this PR is ready for another pass! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This pull request relaxes SQL type checking in
civis.io.civis_file_to_table
, by casting toVARCHAR
when type inconsistency is detected for a given column and at least one input file hasVARCHAR
.CHANGELOG.md
at the repo's root level