-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(rust, python): add HDFS support via hdfs-native package #2612
Conversation
@Kimahriman LGTM! Can you please also add some docs on the integration in the .MD files and if possible links some docs of the possible configs that can be set @rtyler can you also go over it? |
Yeah I agree there should be some, but I didn't see anything for other storage backends so I wasn't sure where to add it. Any recommendation? A new page under |
Yeah under integrations makes sense, perhaps called object storage and then a page for hdfs there @avriiil any inputs on this? Having some small explanation per object store for S3, adls, gcs and mounted storage would make sense, do you want to help on this? |
I added a page. I couldn't get the docs to build locally to verify things though. Just kept getting
|
Yeah there is something broken with the docs for some time now |
0da593f
to
5144485
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this!
Late to the party, I was OOO for a couple of days.
Happy to take a look at creating some more docs if that still makes sense
@ion-elgreco?
~ ~ ~
*Avril Aysha*
DevRel & Content
+351 963681498
…On Fri, 21 Jun 2024 at 07:22, Ion Koutsouris ***@***.***> wrote:
Merged #2612 <#2612> into main.
—
Reply to this email directly, view it on GitHub
<#2612 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQLWMSQ4OSQHETPDAUM7PSLZIPBA7AVCNFSM6AAAAABJSWGJNKVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGIZTQMBSG4YDSNY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@avriiil yeah some docs for each object store would be great 😃 |
sounds good, adding this to my list for next week @ion-elgreco |
Description
Add support for HDFS using hdfs-native, a pure* Rust client for interacting with HDFS. Creates a new
hdfs
sub-crate, adds it as a feature todeltalake
meta crate, and includes it in Python wheels by default. There is a Rust integration test that requires Hadoop and Java to be installed, and makes use of a small Maven program I ship under theintegration-test
feature flag to run a MiniDFS server.*Dynamically loads
libgssapi_krb5
usinglibloading
for Kerberos supportRelated Issue(s)
Resolves #2611
Documentation