Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-33923: [Docs] Tensor canonical extension type specification #33925

Merged
merged 27 commits into from
Mar 15, 2023
Merged
Changes from 4 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
af571cb
Add Fixed size tensor spec to canonical extensions list
AlenkaF Jan 30, 2023
8231150
Apply suggestions from code review
AlenkaF Jan 30, 2023
884d871
Remove implementation-specific metadata
AlenkaF Jan 30, 2023
83edd70
Change order with is_row_major
AlenkaF Jan 30, 2023
16ef6f1
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Jan 30, 2023
4f4ccce
Update metadata part
AlenkaF Jan 30, 2023
92fd7c6
Correct True to true in json
AlenkaF Jan 31, 2023
7873676
Change name from fixed_size_tensor to fixed_shape_tensor
AlenkaF Jan 31, 2023
a4219e3
Add description for ListType parameters
AlenkaF Jan 31, 2023
37e83db
Change the description for ListType parameters
AlenkaF Feb 1, 2023
5c92ff0
Remove is_row_major from the spec
AlenkaF Feb 2, 2023
cb5e2dd
Add dim_names and permutation to optional metadata
AlenkaF Feb 15, 2023
b562b8d
Add notes to the usage of dim_names and permutations metadata
AlenkaF Feb 15, 2023
c44101b
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 15, 2023
24e7c28
Add dim_names and permutation to optional parameters
AlenkaF Feb 15, 2023
333ae67
Add explicit explanation of permutation indices
AlenkaF Feb 15, 2023
4086dfb
Change order with layout
AlenkaF Feb 15, 2023
bd2a515
Rephrase text about absent permutation param
AlenkaF Feb 15, 2023
bc07d7a
Apply suggestions from code review - Joris
AlenkaF Feb 15, 2023
68c6244
Remove redundant sentence in permutations explanation
AlenkaF Feb 16, 2023
3e2bb25
Update value_type description
AlenkaF Feb 22, 2023
a49f14f
Update parameters description
AlenkaF Feb 22, 2023
89d8042
Add a logical layout shape example in the desc of the serialization
AlenkaF Feb 22, 2023
4ff7a65
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 22, 2023
1daf820
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Feb 28, 2023
70059d9
Add note about IPC tensor
AlenkaF Mar 9, 2023
6f44296
Update docs/source/format/CanonicalExtensions.rst
AlenkaF Mar 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion docs/source/format/CanonicalExtensions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,24 @@ same rules as laid out above, and provide backwards compatibility guarantees.
Official List
=============

No canonical extension types have been standardized yet.
Fixed size tensor
=================

* Extension name: `arrow.fixed_size_tensor`.
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved

* The storage type of the extension: ``FixedSizeList``.
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved

* Extension type parameters:

* **value_type** = Arrow DataType of the tensor elements
* **shape** = shape of the contained tensors as a tuple
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved
* **is_row_major** = boolean indicating the order of elements in memory

* Description of the serialization:

The metadata must be a valid JSON object including:

* shape of the contained tensors as an array with key “shape”,
* string defining the order of elements in memory with key “order”.
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved

For example: `{ "shape": [2, 5], "is_row_major": True }`