Is your feature request related to a problem? Please describe.
Auron native Iceberg changelog scan currently supports insert-only changelog tasks.
When a changelog range contains full-data-file delete tasks, Auron falls back to Spark even though this case does not require row-level delete-file handling. The deleted data file can be scanned directly and changelog metadata such as _change_type = DELETE can be materialized from the Iceberg changelog task.
Describe the solution you'd like
Support native execution for Iceberg DeletedDataFileScanTask when:
- the changelog operation is
DELETE
existingDeletes() is empty
Keep the support boundary conservative:
- continue supporting
AddedRowsScanTask with INSERT and empty deletes
- support only full-data-file delete changelog tasks
- keep falling back for row-level deletes, position deletes, equality deletes, update changelog tasks, mixed file formats, and delete tasks with existing delete files
Describe alternatives you've considered
The alternative is to keep falling back to Spark for all delete changelog tasks. That is safe, but it leaves a native execution gap for simple full-data-file delete changelog scans.
Additional context
This change does not attempt to support position deletes, equality deletes, or update changelog operations. Those require separate handling and should be implemented independently.
Is your feature request related to a problem? Please describe.
Auron native Iceberg changelog scan currently supports insert-only changelog tasks.
When a changelog range contains full-data-file delete tasks, Auron falls back to Spark even though this case does not require row-level delete-file handling. The deleted data file can be scanned directly and changelog metadata such as
_change_type = DELETEcan be materialized from the Iceberg changelog task.Describe the solution you'd like
Support native execution for Iceberg
DeletedDataFileScanTaskwhen:DELETEexistingDeletes()is emptyKeep the support boundary conservative:
AddedRowsScanTaskwithINSERTand empty deletesDescribe alternatives you've considered
The alternative is to keep falling back to Spark for all delete changelog tasks. That is safe, but it leaves a native execution gap for simple full-data-file delete changelog scans.
Additional context
This change does not attempt to support position deletes, equality deletes, or update changelog operations. Those require separate handling and should be implemented independently.