You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parquet itself doesn't has a LIST_VIEW type, however, we can read/write a LIST_VIEW inside parquet.
Some notes:
During loading, I think making a arrow::list is ok, and converting a List to ListView is so convenient. So maybe we can first support write
For parquet::arrow::SchemaManifest, maybe it's ok to just store a list as metadata?
TODO:
For writer, we need support PathInfo. The logic is so similiar to origin list node, however, we need support a new kind of RangeSelector. We don't need to support a new list because the writer here doesn't make full use of the contigous array
For schema, we need to handle the parquet/arrow/schema.h well.
Component(s)
C++, Parquet
The text was updated successfully, but these errors were encountered:
@felipecrv@pitrou@wgtmac I'd like to support the write first, becausing reading List and casting to ListView is easy, but casting ListView to List during write is hard. What do you think?
Describe the enhancement requested
Parquet itself doesn't has a
LIST_VIEW
type, however, we can read/write aLIST_VIEW
inside parquet.Some notes:
arrow::list
is ok, and converting a List to ListView is so convenient. So maybe we can first support writeparquet::arrow::SchemaManifest
, maybe it's ok to just store alist
as metadata?TODO:
PathInfo
. The logic is so similiar to origin list node, however, we need support a new kind ofRangeSelector
. We don't need to support a new list because the writer here doesn't make full use of the contigous arrayparquet/arrow/schema.h
well.Component(s)
C++, Parquet
The text was updated successfully, but these errors were encountered: