Spark: apply rewrite manifest action fix to 3.1,3.2 #7296
Conversation
| @@ -77,6 +77,8 @@ public class SparkUtil { | |||
|
|
|||
| private SparkUtil() {} | |||
|
|
|||
| /** @deprecated will be removed in 1.4.0 */ | |||
There was a problem hiding this comment.
Great. Maybe we can put a comment on why, and what the user can call.
Actually, I had a question, that I did not have a chance to ask in original pr: #7263, do we still need to close fileIO, at the end of action? Is that getting handled somewhere?
There was a problem hiding this comment.
That's a good question and it impacts Spark jobs in general, not just this action, thus this is consistent with Iceberg behavior in general AFAIK. There has been some discussion on handling resource cleanup better, @danielcweeks @rdblue do you have any insights into that?
There was a problem hiding this comment.
I updated this and added comments about the deprecation.
There was a problem hiding this comment.
Makes sense. I see the SparkActions takes the table as argument, maybe it could be the responsibility of user then to close it, though it is a bit undocumented.
|
Merged, thanks @bryanck , and others for review |
apache#7263) (apache#7296) This change backports PR apache#7263 to Spark 3.1 and 3.2
apache#7263) (apache#7296) This change backports PR apache#7263 to Spark 3.1 and 3.2
This PR applies the changes from #7263 to Spark v3.2 and v3.1 also. It marks the unused method
SparkUtil.serializableFileIO()as deprecated, as using this to broadcast a FileIO can lead to unintended consequences, i.e. the underlying S3 client being closed during broadcast removal.