Skip to content

[format] Introduce lance file format #5809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

leaves12138
Copy link
Contributor

@leaves12138 leaves12138 commented Jun 27, 2025

Purpose

This pull request $${\color{red} Introduce}$$ $${\color{red}Lance}$$ $${\color{red}File}$$ $${\color{red}Format}$$ to paimon. But pay attention that:

  • This feature is experimental
  • Only support lance V2.1 yet, V2.1 file format in lance is NOT STABLE
  • Please DO NOT use it in production environment

Lance file format only support LocalFileSystem and OssFileSystem in this pull request, Feel free to denote more code for this feature (support more file system).

Please do not merge this pull request, I will do it myself when everything settled down.

Tests

API and Format

Documentation

@leaves12138
Copy link
Contributor Author

This pull request may freeze util lance fix the bug: lancedb/lance#4105 and release a new version

if (!arrowFormatWriter.write(internalRow)) {
flush();
if (!arrowFormatWriter.write(internalRow)) {
throw new RuntimeException("Exception happens while write to orc file");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orc file -> lance file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks~

@leaves12138 leaves12138 changed the title [format] Introduce lance file format [WIP] [format] Introduce lance file format Jul 4, 2025
@leaves12138 leaves12138 changed the title [WIP] [format] Introduce lance file format [format] Introduce lance file format Jul 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants