New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplify and improve expand_data
in Presto
#8233
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8233 +/- ##
==========================================
+ Coverage 65.98% 66.03% +0.04%
==========================================
Files 479 479
Lines 23038 23105 +67
Branches 2552 2552
==========================================
+ Hits 15202 15257 +55
- Misses 7700 7712 +12
Partials 136 136
Continue to review full report at Codecov.
|
|
||
For rows, we return a list of the columns: | ||
|
||
>>> get_children(dict(name="a", type="ROW(BIGINT,FOO VARCHAR)")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more for my understanding than a PR comment. I thought each row has a variable name before the type. For example ROW(FOO BIGINT, BAR VARCHAR)
. When will there be a nameless column again?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question! If you do a query like:
SELECT ARRAY[ROW(100,0,'hello')]
The resulting column has type ARRAY(ROW(INTEGER,INTEGER,VARCHAR(5)))
, without names.
CATEGORY
Choose one
SUMMARY
This PR simplifies the logic of the
expand_data
method in Presto, while making it more generic. Queries that were failing before, likeare now working.
Some of the methods in
db_engine_specs/presto.py
are no longer used and can be removed; I'll do that in a subsequent PR. I wanted to keep this PR simple and small because it's blocking an important release.TEST PLAN
I updated the unit tests, and tested with real data. I'll add more unit tests in the next PR, covering more complex cases.
ADDITIONAL INFORMATION
REVIEWERS
@khtruong