[hive-csv] Infer schema from csv #5267
Conversation
timifasubaa
force-pushed
the
infer_types_for_csv_upload
branch
2 times, most recently
from
7b72ba4
to
adb194b
Compare
superset/db_engine_specs.py
Outdated
| filename = form.csv_file.data.filename | ||
| def convert_to_hive_type(col_type): | ||
| """maps tableschema's types to hive types""" | ||
| if col_type == "number": |
There was a problem hiding this comment.
I think having a dict for the mapping would be cleaner also would allow for people to easily add more types if need be.
There was a problem hiding this comment.
Agree. Added it to config
| @@ -34,6 +34,7 @@ six==1.11.0 | |||
| sqlalchemy==1.2.2 | |||
| sqlalchemy-utils==0.32.21 | |||
| sqlparse==0.2.4 | |||
| tableschema==1.1.0 | |||
There was a problem hiding this comment.
You’ll have to add this to setup.py as well.
timifasubaa
force-pushed
the
infer_types_for_csv_upload
branch
4 times, most recently
from
71b3dcf
to
ec72456
Compare
Codecov Report
@@ Coverage Diff @@
## master #5267 +/- ##
=========================================
- Coverage 61.31% 61.3% -0.02%
=========================================
Files 368 368
Lines 23449 23453 +4
Branches 2713 2713
=========================================
Hits 14378 14378
- Misses 9059 9063 +4
Partials 12 12
Continue to review full report at Codecov.
|
setup.py
Outdated
| @@ -90,6 +90,7 @@ def get_git_sha(): | |||
| 'sqlalchemy', | |||
| 'sqlalchemy-utils', | |||
| 'sqlparse', | |||
| 'tableschema==1.1.0', | |||
There was a problem hiding this comment.
Don't pin in setup.py unless it's strictly required.
superset/config.py
Outdated
| @@ -314,6 +314,14 @@ class CeleryConfig(object): | |||
| # contain all the external tables | |||
| CSV_TO_HIVE_UPLOAD_DIRECTORY = 'EXTERNAL_HIVE_TABLES/' | |||
|
|
|||
| # This contains the mappings from the tableschema types to HIVE types | |||
| TABLESCHEMA_TO_HIVE_TYPES = { | |||
There was a problem hiding this comment.
I think this probably belongs in db_engine_spec. I agree this isn't cut-and-dry as different versions of Hive support different types.
timifasubaa
force-pushed
the
infer_types_for_csv_upload
branch
from
ec72456
to
dde356d
Compare
superset/db_engine_specs.py
Outdated
| bucket_path = app.config['CSV_TO_HIVE_UPLOAD_S3_BUCKET'] | ||
| def convert_to_hive_type(col_type): | ||
| """maps tableschema's types to hive types""" | ||
| TABLESCHEMA_TO_HIVE_TYPES = { |
There was a problem hiding this comment.
Sorry given this no longer resides in the config this should be a lower case variable name.
|
Hm |
timifasubaa
force-pushed
the
infer_types_for_csv_upload
branch
from
dde356d
to
f6ac85b
Compare
(cherry picked from commit b0eee12)
(cherry picked from commit b0eee12)
mistercrunch
added
🍒 0.26.0
🍒 0.26.1
🍒 0.26.2
🍒 0.26.3
🍒 0.27.0
🏷️ bot
Previously all column types were string. This PR leverages the tableschema package to get a more precise column type and uses it in the table creation.
@john-bodley @mistercrunch