ARROW-7622: [Format] Mark Tensor and SparseTensor fields required

[pitrou]

Also make sparse COO indices row-major by default, both by consistency with tensors and because it packs related coordinates sequentially.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7622

[wesm]

Minor comment but otherwise looks good. cc @mrkn for thoughts also

[pitrou]

@mrkn Can you check if those changes look ok to you?

[pitrou]

@mrkn Is there a reason why Tensor strides default to row-major but SparseIndexCOO strides default to column-major?

[mrkn]

Actually, the default strides of SparseIndexCOO is column-major, but it isn't intentionally decided, just historically.
At first, SparseIndexCOO only supported column-major strides because scipy.sparse.coo_matrix always has a column-major index. When I added the support of row-major, I didn't change the default strides. That's all.

I think now is the best timing to change the default strides to row-major.

[pitrou]

(delete a message where I was making a mistake. Row-major is actually the preferred order which makes coordinates sequential in memory)

[pitrou]

I'll note that pydata/sparse uses row-major order for coordinates:

>>> x = np.eye(4)                                                                                                                                                     
>>> x[2, 3] = 5                                                                                                                                                       
>>> s = sparse.COO.from_numpy(x)                                                                                                                                      
>>> s.coords                                                                                                                                                          
array([[0, 1, 2, 2, 3],
       [0, 1, 2, 3, 3]])
>>> s.coords.strides                                                                                                                                                  
(40, 8)

As for scipy.sparse, it only supports 2-dim sparse tensors and stores row and column coordinates in separate arrays... (which has the same memory access properties as column-major)

[pitrou]

Ok, I've switched to row-major by default for COO indices.