[ARROW-210] Add support for large_list and large_string PyArrow DataTypes #191
Conversation
blink1073
changed the title
| expected = Table.from_pydict( | ||
| {"_id": [1, 2], "data": self.expected_times}, | ||
| ArrowSchema([("_id", int32()), ("data", timestamp("ms", tz=tz))]), | ||
| """Test behavior of setting tzinfo CodecOptions in Collection.with_options. |
There was a problem hiding this comment.
What's the context for these datetime test changes?
There was a problem hiding this comment.
I expanded the test. The original wasn't testing what it appeared to. Subleties between timestamps and datetimes.
| @@ -86,3 +86,22 @@ def __eq__(self, other): | |||
| if isinstance(other, type(self)): | |||
| return self.typemap == other.typemap | |||
| return False | |||
|
|
|||
| @classmethod | |||
| def from_arrow(cls, aschema: pa.Schema): | |||
There was a problem hiding this comment.
Should we make the constructor accept pa.Schema rather than introduce a new api method?
There was a problem hiding this comment.
Same same but different. This was quick. I time-boxed my work, and stopped myself short of bigger refactoring. I would love to discuss.
There was a problem hiding this comment.
The difference is that we try to avoid adding new public methods unless there's a clear need.
There was a problem hiding this comment.
We created https://jira.mongodb.org/browse/ARROW-220 after discussing further.
Adding support for two additional Arrow DataTypes:
large_listandlarge_stringappear in theTable.schemawhen one callsPolars to_arrow.This is a small extension of our ListBuilder and StringBuilder classes and additional tests,