Can Iceberg support different FileIO for meta and data?

[johndb2016]

I'd like to write metadata on the local ssd, but data on S3 with Iceberg, and i am wondering whether iceberg supports that.
Should I provide two kinds of FileIO?
Any help willl be appreciated ~

[nastra]

I don't think this is currently supported. In fact we have a potential use case where having this would be helpful and I was planning to work on a prototype for this exact feature this week.

[singhpk234]

[doubt] wouldn't ResolvingFileIO work here ?

This fileIo based on the location decides which FileIO impl should it use and stores, the instantiated fileIo in a local cache to avoid re-instantiation, we can certainly diff paths for meta-data & data

[nastra]

IIUC ResolvingFileIO has the capability to use different schemes, where s3 / s3n / s3a all resolve to S3FileIO and fs resolves to HadoopFileIO. There's no differentiation between which FileIO implemenentation should be used for data and metadata files.

I think the idea behind this ticket is that users would have the capability to explicitly specify which FileIO implementation should be used for data files and which one for metadata files.

[singhpk234]

There's no differentiation between which FileIO implemenentation should be used for data and metadata files.

Agree with you, As per my understanding, wouldn't this differentiation be automatically be done by the location produced for meta-data & data path as :
meta-data path would be : file://mnt/meta-data
data path would be : s3://<bucket>/prefix
(this can be controled via conf's such as write.metadata.path).

Now a user when uses ResolvingFileIO it will implicitly use HadoopFileIO for meta-data (i.e paths like file://mnt/meta-data), S3FileIO for data (i.e paths like s3://<bucket>/prefix).

I think the idea behind this ticket is that users would have the capability to explicitly specify which FileIO implementation should be used for data files and which one for metadata files.

This sounds like an interesting use case as to why would we want two diff FileIO's when let's say both data / meta-data share the same schemes. One case comes to mind that we want to seperate tuned SDK for S3FileIO obj which would read heavy data files and other has light weight meta-data files.

[nastra]

The use case I have in mind for example is: what if users would like to use S3+GCS or simply have different S3 locations for data and metadata files.

[johndb2016]

yep, ResolvingFileIO works for my case, but much better if iceberg provides the capability to set custom FileIO for data & meta individually.