Spark ReadTask is expensive to serialize

[aokolnychyi]

In some Spark jobs, we see a substantial scheduler delay. I assume it happens in TaskSetManager when Spark serializes Iceberg ReadTask. The latter contains a couple of large strings (if you have a lot of columns) and an instance of FileIO (which can contain a full Hadoop conf).

[aokolnychyi]

I can confirm the issue is resolved if we avoid serializing FileIO. The main question is how to achieve that with minimum changes.

[aokolnychyi]

As a short-term solution, we can broadcast EncryptionManager and FileIO in IcebergSource. Then Reader and ReadTask can store references to the broadcasted values and fetch actual ones in createPartitionReader while creating TaskDataReader. This seems to solve the scheduler delay issue.

@rdblue thoughts?

[rdblue]

Using a broadcast sounds good to me for now.

Can you open a PR for this?

[aokolnychyi]

Will open a PR today