Skip to content

Conversation

@Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Mar 24, 2025

Which issue does this PR close?

Part of #1130

What changes are included in this PR?

Add webhdfs based hdfs storage support in iceberg-rust.

Are these changes tested?

Webhdfs service itself is tested at opendal side, maybe we can setup related tests in iceberg-rust too.

Signed-off-by: Xuanwo <github@xuanwo.io>
@liurenjie1024
Copy link
Contributor

Thanks @Xuanwo with this pr. There are two problems with feature based design:

  1. User could only choose hdfs backend at compile time, which seems inflexible to me. For example, I'm integrating iceberg-rust with a compute engine like trino, which may access different storages at runtime.
  2. It may not be transparent to user. IIRC, webhdfs's configuration and url schema are different from native hdfs library.

@liurenjie1024
Copy link
Contributor

Maybe we can write down our choice and discuss at community sync meeting?

@Xuanwo
Copy link
Member Author

Xuanwo commented May 14, 2025

Closing this one since we won't introduce webhdfs support in iceberg-rust for now.

@Xuanwo Xuanwo closed this May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants