API for path generation in Iceberg frameworks

[mccheah]

In the integrations that Iceberg supports out of the box (Spark, Pig), the frameworks decide how to generate paths for written files. However, some sources would prefer to pick their own paths for new files. Some questions for designing such an API include:

• Should this be bundled with the FileIO API?
• Should such a path generation API be concerned about the file's partition values? Partition metadata would be required for the default implementation to maintain existing behavior for Spark.

[mccheah]

@vinooganesh @yifeih @rdblue

[rdblue]

I think this API should pass in the partition tuple. Now that I think about it, the metadata location is determined by the TableOperations implementation, not by FileIO. I think it would make sense to keep those in the same place. So TableOperations would be responsible for determining where to place both metadata and data files, while FileIO would just handle low-level file tasks.

[mccheah]

Ok I'll start taking a look at this now!

[mccheah]

Remark: The paths module has to be a separate entity because once again, we will be instantiating it once on the driver and serializing the instance for executors. I think we can move the metadata file lookup to this module as well.

[rdblue]

@mccheah, in that case is this something that we should add to the FileIO interface? It wouldn't be that much of a stretch to do path selection as well. What do you think?

[mccheah]

Yup I don't have a strong opinion here, so let's bundle it with the FileIO interface. I'll propose a diff but I anticipate wanting to fine tune the exact method signatures as part of the PR review.

[mccheah]

Looks like this is done! I'll close it.

[xabriel]

(For completeness, linking this issue to PR that made it to master: #87)