[Improvement]: Avoid bulk table removal when synchronizing external catalogs and fetching database table listings fails #3533
Description
Search before asking
- I have searched in the issues and found no similar issues.
What would you like to be improved?
Currently when synchronizing the external directory and fetching the database table list fails, the database is considered to contain 0 tables and triggers all its tables to be removed. However, when synchronizing the external directory and fetching the database tables again and succeeds, it triggers the process of adding all the tables. When there are a large number of tables in the database, this can lead to significant thread resource usage and page update delays.
In the logs, we see something like this
2025-04-16 10:39:52,974 INFO [table-explorer-scheduler-0] [org.apache.amoro.server.table.DefaultTableService] [] - Loaded 0 tables from external catalog my_catalog.
2025-04-16 10:39:54,269 INFO [table-explorer-scheduler-0] [org.apache.amoro.server.table.DefaultTableService] [] - Loaded 8307 tables from Amoro server catalog my_catalog.
2025-04-16 10:39:54,271 INFO [table-explorer-executor-24] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_woff_balance_72_20240401(tableId=6661580)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-40] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240406(tableId=6661620)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-14] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_woff_balance_72_20240402(tableId=6661607)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-39] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240407(tableId=6661591)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-17] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240409(tableId=6661603)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-21] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240408(tableId=6661597)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-6] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_woff_balance_72_20240403(tableId=6661592)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-35] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_woff_balance_72_20240404(tableId=6661596)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-1] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240402(tableId=6661576)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-0] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240403(tableId=6661619)
2025-04-16 10:39:54,271 INFO [table-explorer-executor-45] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - Release queue my_catalog with table my_catalog.my_database.mid_acct_balance_76_20240405(tableId=6661611)
......
How should we improve?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Subtasks
No response
Code of Conduct
- I agree to follow this project's Code of Conduct