Support HDFS in Python package

[Kimahriman]

Description

Add support for HDFS in Python now that it's supported in Rust.

Use Case
Using deltalake Python package to read Delta tables in HDFS

Related Issue(s)
#300

[roeap]

Unfortunately, HDFS driver requires JAVA, and one of the core interests is to not depend on that. so closing this as we right now see no way to provide this without java.

[SchutteJan]

@Kimahriman

I see you are working on a Rust implementation for hdfs (No JAVA!) in https://github.com/Kimahriman/hdfs-native and an object store implementation in https://github.com/datafusion-contrib/hdfs-native-object-store

Are you also working on integrating your work into delta-rs? I am experimenting with that myself (https://github.com/SchutteJan/delta-rs) and I am curious what your plans are.

[Kimahriman]

Are you also working on integrating your work into delta-rs? I am experimenting with that myself (https://github.com/SchutteJan/delta-rs) and I am curious what your plans are.

Yeah I do have a branch I've been working on, just been lazy on getting it cleaned up and making a PR. The biggest limitation right now is the libgssapi dependency, which adds a dynamic link and requires some changes to how the wheels would be built. I think I'll try to get the Rust side finished up and make a PR with an option to build a custom Python wheel. I'm currently working on an update to do all gssapi/kerberos things via runtime dynamic loading with libloading. If I can get that working it should simplify building wheels/cross compilation and theoretically could be added to the published wheels if it's accepted.

[Kimahriman]

#2612