Handle individual table failures without failing the entire job #2360
Description
Search before asking
- I searched in the issues and found nothing similar.
Description
Description
Currently, the Tiering Service processes multiple tables for data tiering. However, if a task for a specific table fails due to non-recoverable reasons (e.g., AccessDeniedException for remote storage, missing Paimon Catalog permissions, or metadata corruption), the entire Tiering Service job might crash or enter a restart loop.
Consistent with our implementation in the Committer node, we should improve the fault tolerance of the Tiering Service.
Proposed Changes
The Tiering Service should be decoupled from individual table task failures:
Isolation of Failures: If a tiering task for Table A fails due to an exception (especially permission-related or configuration-related), the service should catch the exception and prevent it from propagating to the main thread.
Table Status Marking: Mark the specific table as "Failed" or "Error" in memory (and potentially log the state).
Continue with Next Table: The service should continue processing the next available tables (Table B, Table C, etc.) in the queue instead of stopping the entire job.
The TieringReader should send to TieringEnumerator and the TieringEnumerator should notify this to all TieringReader and Committer so that they can clear their status for this failed table.
Willingness to contribute
- I'm willing to submit a PR!