Support for Lance in LakeSail

[jonasdedden]

Sail (https://github.com/lakehq/sail) is a drop-in replacement for Spark, offering up to ~30x more performance and also tremendously better resource usage statistics compared to Spark.

What do you think about integration of Lance support for Sail? I opened an issue here, with some vibecoded rough PoC: lakehq/sail#2573

[Xuanwo]

Of course it's a good idea!

[jonasdedden]

Just for documentation, I implemented a Lance plugin for Sail here. It requires some upstream changes in Sail to be merged first though.

Sail's maintainers pointed out that they currently see some problem with their Python datasource plugins, and I guess one would have to publish this Lance plugin for Sail separately on PyPI or so, probably under the name sail-lance. In principle it also could live somewhere in the lance-format namespace, I don't care 🤷‍♂️ Maybe there are technical reasons to do one or the other.