fix(spark): make export_instants descending order work#19172
Conversation
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the contribution! This PR fixes export_instants(..., desc => true), which previously always threw UnsupportedOperationException because Collections.reverse was called on read-only lists produced by asJava over immutable Scala collections; it now wraps both the instant and archive-status lists in mutable ArrayLists and adds regression test coverage. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
export_instants(..., desc => true) threw UnsupportedOperationException on every call: the instant/status lists were built via asJava over immutable Scala collections (read-only java.util.List), and the desc branch reverses them in place with Collections.reverse, which calls List.set. Wrap both in a mutable ArrayList. Adds TestExportInstantsProcedure coverage including the desc-ordering regression case.
f3fee10 to
ec33b4d
Compare
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR fixes export_instants(..., desc => true), which previously threw UnsupportedOperationException because Collections.reverse was called on read-only lists produced by asJava over immutable Scala collections. Wrapping both nonArchivedInstants and archivedStatuses in mutable ArrayList copies resolves the issue, and the ascending path is unaffected. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
Describe the issue this Pull Request addresses
export_instants(..., desc => true)throwsUnsupportedOperationExceptionon every invocation.ExportInstantsProcedurebuilds its instant and file-status lists viaasJavaover immutable Scala collections, producing read-onlyjava.util.Lists. Thedescbranch then reverses them in place withCollections.reverse, which callsList.setand fails on a read-only list. So descending export has never worked.Summary and Changelog
nonArchivedInstantsandarchivedStatusesin a mutablenew java.util.ArrayList(...)soCollections.reversesucceeds. Ascending export is unaffected.TestExportInstantsProcedurecoverage: ascending export, the descending-order regression case (which fails without this fix), action filtering, and the invalid-local-folder error path.Impact
Fixes
export_instantswithdesc => true. No change to ascending behavior or other procedures.Risk Level
low. One-line-per-list change from an immutable wrapper to a mutable copy; the descending regression test exercises the previously-broken path.
Documentation Update
none
Contributor's checklist