Feature: support scalar indices on nested columns

[westonpace]

SDK

Python

Description

LanceDb has strong support for nested fields as they are quite common in ML/AI workloads. However, it is not possible today to create a scalar index on a nested field. We should add support for that.

[westonpace]

Inspired by #899 (comment)

[BTheunissen]

This would be great. Would PyArrow list operations such as array_has_any be able to take advantage of the scalar indices in this case?

[westonpace]

By "nested fields" I was mostly thinking of struct fields. A nested list field would require some more work. The btree implementation is expecting each node to be a (value, row_address) tuple.

Once you have a list field you have more than one value per row and that assumption breaks.

I had thought about this in the past. If your value set is somewhat bounded (e.g. less than tens of thousands of possible values) then a bitmap index could be used for array_has_any.