This repository was archived by the owner on Nov 1, 2024. It is now read-only.
Refactor DType definition into dtypes_core.py #468
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
CLA Signed
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D38358389 |
wenleix
added a commit
to wenleix/torcharrow
that referenced
this pull request
Aug 2, 2022
Summary: Pull Request resolved: pytorch#468 dtypes.py contains of two parts: (1) Standard Arrow-compatible DataType definition (Int8/16/32/64, Float32/64, String, List, Map, Struct) (2) Utility functions around DType The DType definition is quite standalone and stable; Refactor DataType definition into seperate file so allows reuse in next gen TorchArrow, such as TorchArrow-UPM or Tensor-based TA In theory we could let dtypes.py only contain DType definition, and move other things into `dtype_util.py`. Starting with this first step. Differential Revision: D38358389 fbshipit-source-id: d855fcd00ad9ff2dc018832038bf79a75d698d47
wenleix
force-pushed
the
export-D38358389-to-fbsync
branch
from
19b8cc3 to
0583471
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D38358389 |
wenleix
commented
Aug 3, 2022
|
|
||
| @dataclass(frozen=True) | ||
| class List(DType): | ||
| item_dtype: DType |
Contributor
Author
There was a problem hiding this comment.
Here we use item_dtype and fixed_size while in PyArrow it's called value_type and list_size: https://arrow.apache.org/docs/python/generated/pyarrow.list_.html
Doesn't feel strong about fixed_size vs. list_size . I guess item can be renamed to value in future diffs :). (since ListColumn also contains values and offsets.
Contributor
Author
There was a problem hiding this comment.
or, element_dtype as per Velox and Trino/Presto:
wenleix
commented
Aug 3, 2022
| @dataclass(frozen=True) | ||
| class Map(DType): | ||
| key_dtype: DType | ||
| item_dtype: DType |
Contributor
Author
There was a problem hiding this comment.
I think this key/item is coming from PyArrow's Map Type: https://arrow.apache.org/docs/python/generated/pyarrow.map_.html
But in many columnar Database (Presto/Velox) calls it key/value instead. key/value also seems to align with Pythonic API (in Python Dict, item means the (key, value) pair).
Summary: Pull Request resolved: pytorch#468 dtypes.py contains of two parts: (1) Standard Arrow-compatible DataType definition (Int8/16/32/64, Float32/64, String, List, Map, Struct) (2) Utility functions around DType The DType definition is quite standalone and stable; Refactor DataType definition into seperate file so allows reuse in next gen TorchArrow, such as TorchArrow-UPM or Tensor-based TA In theory we could let dtypes.py only contain DType definition, and move other things into `dtype_util.py`. Starting with this first step. Reviewed By: dracifer Differential Revision: D38358389 fbshipit-source-id: 243e93bece169c91ccdd6a8ea3d6ac4d087b078f
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D38358389 |
wenleix
force-pushed
the
export-D38358389-to-fbsync
branch
from
0583471 to
41f6f3e
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Aug 6, 2022
Summary: Pull Request resolved: #468 dtypes.py contains of two parts: (1) Standard Arrow-compatible DataType definition (Int8/16/32/64, Float32/64, String, List, Map, Struct) (2) Utility functions around DType The DType definition is quite standalone and stable; Refactor DataType definition into seperate file so allows reuse in next gen TorchArrow, such as TorchArrow-UPM or Tensor-based TA In theory we could let dtypes.py only contain DType definition, and move other things into `dtype_util.py`. Starting with this first step. Reviewed By: dracifer Differential Revision: D38358389 fbshipit-source-id: 632037498290222a60794e26d0db0c98c73f1391
wenleix
added a commit
that referenced
this pull request
Aug 6, 2022
Summary: Pull Request resolved: #468 dtypes.py contains of two parts: (1) Standard Arrow-compatible DataType definition (Int8/16/32/64, Float32/64, String, List, Map, Struct) (2) Utility functions around DType The DType definition is quite standalone and stable; Refactor DataType definition into seperate file so allows reuse in next gen TorchArrow, such as TorchArrow-UPM or Tensor-based TA In theory we could let dtypes.py only contain DType definition, and move other things into `dtype_util.py`. Starting with this first step. Reviewed By: dracifer Differential Revision: D38358389 fbshipit-source-id: 632037498290222a60794e26d0db0c98c73f1391
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
dtypes.py contains of two parts:
(1) Standard Arrow-compatible DataType definition (Int8/16/32/64, Float32/64, String, List, Map, Struct)
(2) Utility functions around DType
The DType definition is quite standalone and stable now; Refactor DataType definition into seperate file so allows reuse in next gen TorchArrow, such as TorchArrow-UPM or Tensor-based TA
In theory we could let dtypes.py only contain DType definition, and move other things into
dtype_util.py. Starting with this first step.Differential Revision: D38358389