[HUDI-9653] Add Spark Procedures for showing requested, completed cleans

[vamshikrishnakyatham]

What is the purpose of the pull request

This PR implements new SQL procedures for viewing Hudi cleaning operations to provide comprehensive visibility into table maintenance activities:

• show_cleans: Display completed clean operations with timing and file deletion statistics
• show_clean_plans: Display clean plans (both requested and completed) with retention policies and state information
• show_cleans_metadata: Display partition-level clean metadata with detailed statistics
• Enhanced with showArchived parameter: Access both active and archived timeline data

These procedures help users monitor cleaning performance across the entire timeline history, debug storage issues with precise filtering, and understand table maintenance operations using familiar SQL syntax.

Brief change log

• Implement ShowCleansProcedure for displaying completed clean operations
• Implement ShowCleansPlanProcedure for displaying clean plans with state information (REQUESTED, INFLIGHT, COMPLETED)
• Add ShowCleansPartitionMetadataProcedure for partition-level clean metadata
• • Support limit parameter for result pagination
• Add showArchived parameter to all procedures for accessing archived timeline data
• Register new procedures in HoodieProcedures.scala
• Add comprehensive test suite TestShowCleansProcedures.scala
• Include proper error handling for non-existent tables and invalid filter expressions
• Follow existing procedure patterns and coding standards

Change Logs

Context: Added three new SQL procedures to provide comprehensive visibility into Hudi's cleaning operations, including historical data from archived timelines, which was previously only available through CLI tools or direct timeline inspection.

Summary:

• New show_cleans procedure displays completed cleaning operations with metadata like timing, files deleted, and retention policies
• New show_clean_plans procedure shows clean operations in all states (REQUESTED, INFLIGHT, COMPLETED) with state field
• New show_cleans_metadata procedure provides partition-level cleaning details for debugging
• showArchived parameter: Access both active and archived timeline data using metaClient.getArchivedTimeline.mergeTimeline(metaClient.getActiveTimeline)
• All procedures follow existing Hudi procedure patterns and include comprehensive error handling

Impact

Public API Changes:

• Adds three new SQL procedures: show_cleans, show_clean_plans, show_cleans_metadata
• All procedures accept optional parameters:
  • limit parameter for pagination
  • showArchived parameter for accessing archived timeline data
• New procedures are registered and available via CALL statements in Spark SQL

User-Facing Features:

• Users can now monitor cleaning operations directly via SQL across entire timeline history
• Active data access: call show_clean_plans(table => 'my_table', limit => 5)
• Archived data access: call show_cleans(table => 'my_table', showArchived => true, limit => 5)
• Better debugging capabilities for storage and maintenance issues
• Consistent interface with other Hudi procedures

Performance Impact:

• Minimal performance impact - procedures only read timeline metadata
• No impact on write operations or table performance
• Procedures use existing timeline APIs with no additional I/O overhead
• Archived timeline access may have slight performance cost for large archived datasets

Risk level (write none, low medium or high below)

Low

This is a new feature addition that only adds SQL procedures without modifying existing functionality. The changes are isolated, well-tested, and follow established patterns. No existing APIs or behaviors are modified. The SQL filter expressions use Spark's built-in SQL parser for safety and consistency.

Documentation Update

Website Update Required:

• New SQL procedures need to be documented on Hudi website under SQL procedures section
• Will create follow-up JIRA ticket for website documentation update
• Code includes comprehensive ScalaDoc documentation for all new classes and methods

Config Changes:

• None - no new configurations added or existing defaults changed

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable including showArchived and filtering capabilities
• CI passed

[rahil-c]

@vamshikrishnakyatham can you check github ci when you get a chance for issues.

Error:  /home/runner/work/hudi/hudi/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowCleansPlanProcedure.scala:83: error: type mismatch;
Error:   found   : scala.collection.mutable.Buffer[org.apache.hudi.common.table.timeline.HoodieInstant]
Error:   required: Seq[org.apache.hudi.common.table.timeline.HoodieInstant]
Error:      cleanInstants.sortWith((a, b) => comparator.compare(a, b) < 0)
Error:                            ^

[rahil-c]

@yihua @nsivabalan @dannyhchen @jonvex I did an initial pass of the PR and the core logic looks good to me. Was wondering if I can have a committer take a look as well since I will not have the ability to merge it.

[vinothchandar]

thanks for the review @rahil-c .. Waiting on CI.

Good job , @vamshikrishnakyatham !

[rahil-c]

@vamshikrishnakyatham I think the failures on the CI are likely flaky test failures. You will need to either ask one of the hudi commmitters to rerun the failed jobs or push some other change to trigger ci.

[vinothchandar]

rekicked CI

[rahil-c]

Thanks @vamshikrishnakyatham for this change! I have approved from my side.

[hudi-bot]

CI report:

• dac3822 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build