[FEAT] [Tensor] Add support for Tensor and FixedShapeTensor types.

[clarkzinzow]

This PR adds basic support for Tensor and FixedShapeTensor types, allowing us to represent arbitrary n-dimensional arrays in dataframe columns.

Heterogeneous/ragged tensor type

    data = [
        np.arange(8,).reshape((2, 2, 2)),
        np.arange(8, 32).reshape((2, 2, 3, 2)),
        None,
    ]
    s = Series.from_pylist(data, pyobj="force")
    s = s.cast(DataType.tensor(DateType::Int64))

Homogeneous/fixed-shape tensor type

    shape = (3, 2, 2)
    data = [
        np.arange(12, dtype=np_dtype).reshape(shape),
        np.arange(12, 24, dtype=np_dtype).reshape(shape),
        None,
    ]
    s = Series.from_pylist(data, pyobj="force")
    s = s.cast(DataType.tensor(DateType::Int64, shape))

TODOs

• DataFrame-level test coverage.
• Add repr for tensor types.
• Convert NumPy ndarrays to Tensor type on Python ingress (Series.from_pylist()).
• Convert Tensor type to NumPy ndarrays on Python egress (Series.to_pylist()).
• Casting between tensor types.
• Casting to/from image types.
• Convert pyarrow fixed_shape_tensor extension type to FixedShapeTensor type on Arrow ingress.
• Convert FixedShapeTensor type to pyarrow fixed_shape_tensor extension type on Arrow egress.
• Casting to/from embedding type.
• Add some basic kernels that delegate to the ndarray crate.

[codecov]

Codecov Report

Merging #1073 (765bc53) into main (03a6646) will increase coverage by 0.03%.
The diff coverage is 92.72%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1073      +/-   ##
==========================================
+ Coverage   88.39%   88.42%   +0.03%     
==========================================
  Files          54       54              
  Lines        5488     5555      +67     
==========================================
+ Hits         4851     4912      +61     
- Misses        637      643       +6     

Impacted Files	Coverage Δ
daft/datatype.py	92.01% <86.66%> (-0.71%)	⬇️
daft/series.py	92.09% <89.74%> (-0.44%)	⬇️
daft/dataframe/dataframe.py	88.84% <100.00%> (ø)
daft/runners/partitioning.py	82.35% <100.00%> (ø)
daft/runners/ray_runner.py	91.66% <100.00%> (ø)
daft/table/table.py	88.46% <100.00%> (+1.01%)	⬆️
daft/udf.py	94.11% <100.00%> (ø)
daft/utils.py	86.04% <100.00%> (+1.83%)	⬆️

[samster25]

LGTM! I'm looking forward to factoring our Monolithic casting logic when we get a chance

[samster25]

should these match?

[clarkzinzow]

Hmm what do you mean by "match", and to what are you referring to by "these"? The first two fields are the data array and the corresponding offsets array, where the former has a native templated type and the latter should always have an i64 type.

The last two fields are the shape array and the corresponding offsets array.