Parallelize metadata and data file removal on snapshot expiration?

[fbocse]

Any particular reason for which metadata and data file removal tasks aren't parallelized using thread pool? https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L319-L341

[rdblue]

It's probably safe to parallelize this, but I think we didn't do it originally because we don't want to get "slow down" responses from services like S3. Seems like adding an option to the API here would be a good idea. Maybe allow passing in a thread pool for deletes?

[fbocse]

Oh I see, so throttling may occur if we parallelize these delete operations.
@rdblue How about looking at other fs functions to delete maybe like fs.deleteOnExit instead of ops.io().deleteFile(file)?
Sounds like it's some sort of lazy delete file API that may work around throttling issues?

I could look into what it's implementation looks like for S3 and ADLS but I think that this API would work better for this particular case. wdyt?

[rdblue]

I don't think we want to use deleteOnExit. That just queues the delete to happen in a shutdown hook, but I don't think there are many guarantees about reliability. That would make it so we no longer retry failed operations and might affect recovery when some, but not all of them fail. It also wouldn't help with parallelism because it would happen in a single shutdown hook thread.

I think the best way to parallelize this operation is to run in a thread pool passed in by the user. Then the user could configure the parallelism of that pool to ensure that it is not so high that it hits throttling exceptions.

[fbocse]

You're right, lack of consistency guarantees would make things complicated and eventually flawed.
Your suggestion on allowing user to pass in a configurable thread pool should help w/ parallelizing this effort.

[aokolnychyi]

Seems like this can be resolved?