[WIP] Add support for periodically refreshing dynamic catalogs from the CatalogStore

[ppaglilla]

Description

Proposal to add support for Trino installations using dynamic catalogs to periodically refresh catalogs from the configured CatalogStore. It allows Trino to periodically list the catalogs available in the CatalogStore and add/remove/update the available catalogs accordingly.

The objective of this proposed change is to allow administrators to externally manage dynamic catalogs and have Trino periodically sync to their configuration, rather than needing to push catalog changes to Trino through CREATE/DROP CATALOG statements.

Catalog refreshes are disabled by default and can be enabled through the catalog.refresh.enabled configuration property. The refresh interval defaults to 60 seconds and can be configured through the catalog.refresh.refresh-interval

Example configuration:

catalog.management=dynamic
catalog.refresh.enabled=true
catalog.refresh.refresh-interval=60s
catalog.store=my_catalog_store

I would love to hear feedback on whether this change is welcome and on the approach itself. I would also appreciate any recommendations on how to effectively test the change.

Additional context and related issues

Catalog refreshes are implemented in the CoordinatorDynamicCatalogManager::refreshCatalogs() method. The approach is to list the catalogs from the catalog store, compare them to the currently active catalogs and determine the catalogs that were added, removed or updated. With that information, update the catalog state accordingly.

Catalog refreshing is triggered by the CatalogRefreshTask, which works very similarly to the existing CatalogPruneTask.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

[cla-bot]

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

[ebyhr]

@cla-bot check

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!

[github-actions]

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.

[github-actions]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.