-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[format] Introduce lance file format #5809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
77881eb
to
fb4ff90
Compare
This pull request may freeze util lance fix the bug: lancedb/lance#4105 and release a new version |
if (!arrowFormatWriter.write(internalRow)) { | ||
flush(); | ||
if (!arrowFormatWriter.write(internalRow)) { | ||
throw new RuntimeException("Exception happens while write to orc file"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
orc file -> lance file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks~
07257fe
to
4e19f79
Compare
4e19f79
to
3f4bc83
Compare
6825189
to
8c04295
Compare
paimon-format-lance/pom.xml
Outdated
<artifactId>*</artifactId> | ||
</exclusion> | ||
</exclusions> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this before provided dependencies.
JarJniLoader.loadLib(LanceFileFormatFactory.class, "/nativelib", "crypto"); | ||
JarJniLoader.loadLib(LanceFileFormatFactory.class, "/nativelib", "ssl"); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need this class anymore
paimon-format-lance/pom.xml
Outdated
<version>1.3-SNAPSHOT</version> | ||
</parent> | ||
|
||
<artifactId>paimon-format-lance</artifactId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
paimon lance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
Purpose
This pull request$${\color{red} Introduce}$$ $${\color{red}Lance}$$ $${\color{red}File}$$ $${\color{red}Format}$$ to paimon. But pay attention that:
Lance file format only support LocalFileSystem and OssFileSystem in this pull request, Feel free to denote more code for this feature (support more file system).
Please do not merge this pull request, I will do it myself after everything settled down.
Tests
API and Format
Documentation