New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support delimiters in hive csv upload #9971
feat: support delimiters in hive csv upload #9971
Conversation
Codecov Report
@@ Coverage Diff @@
## master #9971 +/- ##
==========================================
- Coverage 68.90% 68.89% -0.01%
==========================================
Files 584 584
Lines 31055 31056 +1
Branches 3180 3180
==========================================
- Hits 21397 21396 -1
- Misses 9549 9551 +2
Partials 109 109
Continue to review full report at Codecov.
|
4d93b9c
to
26201b4
Compare
sql = f"""CREATE TABLE {str(table)} ( {schema_definition} ) | ||
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS | ||
TEXTFILE LOCATION '{location}' | ||
sql = text( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note text
is used for language agnostic :param
s.
superset/db_engine_specs/hive.py
Outdated
engine.execute(sql) | ||
engine.execute( | ||
sql, | ||
delim=csv_to_df_kwargs['sep'].encode().decode('unicode_escape'), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Required to escape \\t
as \t
.
@bkyryliuk Used params as suggested. Switched back to the original row format (single character delimiter) because unicode made it hard to use complex delimiters. |
ping @bkyryliuk |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@serenajiang looks good, please fix the linter
d6a0571
to
b10fb8a
Compare
Co-authored-by: serena-jiang <serena.jiang@airbnb.com>
SUMMARY
Previously, hive csv upload only supported
,
as a delimiter. This is confusing for users because the UI allows you to specify a delimiter.Altered to allow other delimiters.
TEST PLAN
Tested csv + tsv uploads.
ADDITIONAL INFORMATION
@etr2460 @bkyryliuk @john-bodley