Error writing Lance dataset

[changhiskhan]

first reported by @cemoody (on discord)

Lance version: v0.4.2, v0.4.3

When writing this dataset:

lance.write_dataset(combined.select(['id', 'vector']), lance_dir)

An error is raise:

thread '<unnamed>' panicked at 'assertion failed: (offset + length) <= self.len()', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow-data-33.0.0/src/data.rs:543:9
*** pyo3_runtime.PanicException: assertion failed: (offset + length) <= self.len()

However, writing each column individually works fine:

(Pdb) lance.write_dataset(combined.select(['id']), lance_dir / '3')
<lance.dataset.LanceDataset object at 0x7f4b8c15ffa0>
(Pdb) lance.write_dataset(combined.select(['vector']), lance_dir / '4')
<lance.dataset.LanceDataset object at 0x7f4b8c15fe20>

Workaround:

First reading the underlying data via pandas, then converting that to PyArrow.