branch-4.0: [Feature](Iceberg) Implement expire_snapshots procedure for Iceberg tables #59979 #60647
+533
−107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
suxiaogang223
commented
Feb 10, 2026
suxiaogang223
commented
- Cherry-picked from [Feature](Iceberg) Implement expire_snapshots procedure for Iceberg tables #59979
…ables (apache#59979) - Issue Number: apache#58199 This PR implements the `expire_snapshots` procedure for Iceberg tables, following the Apache Iceberg Spark procedure specification. This procedure removes old snapshots from Iceberg tables to free up storage space and improve metadata performance. - **File:** `fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/action/IcebergExpireSnapshotsAction.java` - Implemented `executeAction()` method to expire snapshots using Iceberg's `ExpireSnapshots` API - Added `getResultSchema()` method returning 6-column output matching Spark's schema - Added `parseTimestamp()` helper method to support ISO datetime and milliseconds formats - Updated validation to allow `snapshot_ids` as a standalone parameter - Fixed `retain_last` behavior: when specified alone, automatically sets `expireOlderThan` to current time | Parameter | Description | |-----------|-------------| | `older_than` | Timestamp before which snapshots will be removed (ISO datetime or milliseconds) | | `retain_last` | Number of ancestor snapshots to preserve | | `snapshot_ids` | Comma-separated list of specific snapshot IDs to expire | | `max_concurrent_deletes` | Size of thread pool for delete operations | | `clean_expired_metadata` | When true, cleans up unused partition specs and schemas | The procedure returns 6 columns: - `deleted_data_files_count` - `deleted_position_delete_files_count` - `deleted_equality_delete_files_count` - `deleted_manifest_files_count` - `deleted_manifest_lists_count` - `deleted_statistics_files_count` - **File:** `regression-test/suites/external_table_p0/iceberg/action/test_iceberg_execute_actions.groovy` - Added functional tests for `expire_snapshots` with `retain_last` parameter - Added validation tests for `snapshot_ids` parameter - Updated error message expectations ```sql -- Expire snapshots, keeping only the last 2 ALTER TABLE catalog.db.table EXECUTE expire_snapshots("retain_last" = "2"); -- Expire snapshots older than a specific timestamp ALTER TABLE catalog.db.table EXECUTE expire_snapshots("older_than" = "2024-01-01T00:00:00"); -- Expire specific snapshots by ID ALTER TABLE catalog.db.table EXECUTE expire_snapshots("snapshot_ids" = "123456789,987654321"); -- Combine parameters ALTER TABLE catalog.db.table EXECUTE expire_snapshots("older_than" = "2024-06-01T00:00:00", "retain_last" = "5"); ``` (cherry picked from commit 6d9883e)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
morningman
approved these changes
Feb 10, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.