Proposal to refactor some large individual files into modules

[sdd]

A few of the source files within iceberg-rust are getting very large (especially manifest.rs, manifest_list.rs, schema.rs, and table_metadata.rs). Maybe it's just my taste but I find it painful to navigate when they get much over 1000 lines.

How would people feel about some PRs to turn these into module folders with nested files so that things like the serde code, builders, visitors etc can live in separate files in each subfolder, keeping the main mod.rs file focussed on the core type definitions & logic?

[marvinlanhenke]

[...]with nested files so that things like the serde code, builders, visitors etc can live in separate files in each subfolder

I like the idea especially regarding the serde code. Other than that I think I can still navigate quite quickly with the structure we have atm.

Since a lot of features are still missing in terms of read support and a writer framework has not been implemented - I'd wait for those changes until I can reason better about a suitable structure / subfolders?

However, for the serde code I think it be appropriate to do it right now - since it's very limited in scope.

[liurenjie1024]

In general I prefer smaller files, but I agree with @marvinlanhenke that for we can wait for more features to land before conducting further restructure. Maybe we can start with serde part and change them one by one so that we don't have too much conflicts?

cc @Xuanwo @ZENOTME @Fokko What do you think?

[Xuanwo]

In general I prefer smaller files, but I agree with @marvinlanhenke that for we can wait for more features to land before conducting further restructure.

I feel the same way. It's too soon for us to undertake such refactors.