-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add union to test_all_types, and arrow and json R/W #7701
Add union to test_all_types, and arrow and json R/W #7701
Conversation
unsupported arrow type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! I have left some comments about the JSONTransform
implementation.
I'm also wondering about how other systems map from JSON to UNION. Do you have any idea? Perhaps we're the only system out there that does this. I always thought that reading JSON like this:
{"value": 42}
{"value": [1, 2, 3]}
Could be read using UNION(int INTEGER, int_list INT[])
without having to wrap them in an object with the keys "int" or "int_list".
But now that I see your implementation, I'm not sure what makes the most sense, especially since I'm not a heavy JSON user myself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! Looks great.
As this is adding unions to the arrow conversion - could we add some union-specific arrow conversion tests in the Python client? In particular nested unions, unions with lists/structs inside them, unions inside lists/structs, all are not covered yet but could potentially cause problems.
…x/python-union-description
d83de13
to
0791ba5
Compare
…/python-union-description
Failures don't look related my changes? |
Could you merge with master? |
…/python-union-description
👏👏👏 |
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
- Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state: [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning - Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz: Python TIMESTAMPTZ support - Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci: [CI] More CI reduction and clean-up - Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description: Add union to test_all_types, and arrow and json R/W - Merge pull request duckdb/duckdb#8497 from samansmink/pending-execute-result-api-change: Add PendingExecutionResult::ALL_TASKS_BLOCKED
- Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state: [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning - Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz: Python TIMESTAMPTZ support - Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci: [CI] More CI reduction and clean-up - Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description: Add union to test_all_types, and arrow and json R/W - Merge pull request duckdb/duckdb#8497 from samansmink/pending-execute-result-api-change: Add PendingExecutionResult::ALL_TASKS_BLOCKED
This started out adding support to the Python
.description
field for the union data type, but I realised that we didn't have it in thetest_all_types
table function.uint8_t
instead of aint8_t