Tracking issues to make iceberg-rust more friendly to Rust users

[Xuanwo]

This issue is used to track issues to make iceberg-rust more friendly to Rust users.

Context

Iceberg-rust itself feels too much like Java.

It’s partly the history of how iceberg-rust was decided. The early core group of contributors consisted of two kinds of people: one group familiar with Iceberg but not yet experienced in Rust, who wanted to learn Rust by contributing to iceberg-rust; the other group familiar with Rust but knowing very little about Iceberg, who wanted to learn more about Iceberg through contribution.

So in either case, following the same pattern from Java wasn’t a bad option. Otherwise, we wouldn’t have been able to build iceberg-rust into its current shape.

The other reason, as @twuebi and @liurenjie1024 mentioned, is that although Iceberg has a great spec and the spec intends to be language-agnostic and allow different languages to have their own implementations, that never truly happened. In reality, many things aren’t defined or clearly spelled out in the Iceberg spec, so we have to refer to Iceberg-Java’s current behavior to make things work.

I’m not here to defend ourselves. It’s just the reality we’re facing. To improve in this area, we need to find a way to make our implementation correct and gradually improve it to be more friendly to Rust users.

Tasks

Please provide your feedback here.

[manuzhang]

Do you have an example that is unfriendly to Rust users (BTW, developers sound better than users to me)?

[mdub]

I just (this week) started using iceberg-rust in a project. So, these are initial impressions.

Firstly, many thanks for making and maintaining this! The foundations are solid, and I've been able to get something going relatively quickly. It's also been gratifying to see the activity in both Issues and PRs - this project obviously has an active and interested user-base.

To be honest, I'm pretty new to Rust as well, but I can see how patterns have been carried across from the world of Java (and Spark), in a way that seems slightly awkward.

One example I've come up against is configuration. I'm building a CLI, and focusing on AWS-centric use-cases at the moment, i.e. GlueCatalog, and S3 storage. It feels quite awkward having to construct GlueCatalog and FileIO by providing props: HashMap<String, String>. For my use-cases, I would prefer being able to provide an aws_config:SdkConfig (or equivalent) to provide common config for Glue and S3 interactions (region, credentials).