DeltaLake: ClickHouse Server fails to start if the Bucket does not exist.

[mgill25]

Version: 23.9.2.46708

Describe what's wrong

A clear and concise description of what works not as it is supposed to.

The following exception causes the server to crash.

2024.01.09 12:09:41.101670 [ 85 ]  {} <Error> Application: DB::Exception: Could not list objects in bucket 'bucket-name' with key '<redacted>, S3 exception: `NoSuchBucket`, message: 'The specified bucket does not exist': Cannot attach table `default`.`<table_name>` from metadata file /var/lib/clickhouse/store/.../.../table_name.sql from query ATTACH TABLE default.table_name UUID '<uuid>' (...) ENGINE = DeltaLake('https://s3.amazonaws.com/path/to/delta/lake', 'aws_access_key_id', 'aws_secret_access_key')
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c7a6957 in /usr/bin/clickhouse
1. DB::S3Exception::S3Exception<String, String, String, String>(Aws::S3::S3Errors, fmt::v8::basic_format_string<char, fmt::v8::type_identity<String>::type, fmt::v8::type_identity<String>::type, fmt::v8::type_identity<String>::type, fmt::v8::type_identity<String>::type>, String&&, String&&, String&&, String&&) @ 0x0000000010eabe88 in /usr/bin/clickhouse
2. DB::S3DataLakeMetadataReadHelper::listFiles(DB::StorageS3::Configuration const&, String const&, String const&) @ 0x0000000010eaba39 in /usr/bin/clickhouse
3. DB::DeltaLakeMetadataParser<DB::StorageS3::Configuration, DB::S3DataLakeMetadataReadHelper>::getFiles(DB::StorageS3::Configuration const&, std::shared_ptr<DB::Context const>) @ 0x00000000071aabce in /usr/bin/clickhouse
4. DB::IStorageDataLake<DB::StorageS3, DB::StorageDeltaLakeName, DB::DeltaLakeMetadataParser<DB::StorageS3::Configuration, DB::S3DataLakeMetadataReadHelper>>::getDataFiles(DB::StorageS3::Configuration const&, std::shared_ptr<DB::Context const>) @ 0x0000000010b8846b in /usr/bin/clickhouse
5. DB::IStorageDataLake<DB::StorageS3, DB::StorageDeltaLakeName, DB::DeltaLakeMetadataParser<DB::StorageS3::Configuration, DB::S3DataLakeMetadataReadHelper>>::getConfigurationForDataRead(DB::StorageS3::Configuration const&, std::shared_ptr<DB::Context const>, std::vector<String, std::allocator<String>> const&) @ 0x0000000010b87cd3 in /usr/bin/clickhouse
6. std::shared_ptr<DB::IStorage> std::__function::__policy_invoker<std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerStorageDeltaLake(DB::StorageFactory&)::$_0, std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::StorageFactory::Arguments const&) (.llvm.872024380460091824) @ 0x0000000012d935ac in /usr/bin/clickhouse
7. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x00000000129aa1bb in /usr/bin/clickhouse
8. DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, bool) @ 0x00000000116901d0 in /usr/bin/clickhouse
9. DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x00000000116af08f in /usr/bin/clickhouse
10. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::TablesLoader::startLoadingTables(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, std::shared_ptr<DB::Context>, std::vector<DB::StorageID, std::allocator<DB::StorageID>> const&, unsigned long)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000120b96ff in /usr/bin/clickhouse
11. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000c8927bf in /usr/bin/clickhouse
12. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c8962dc in /usr/bin/clickhouse
13. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c894b07 in /usr/bin/clickhouse
14. ? @ 0x00007fc8f0cf7ac3 in ?
15. ? @ 0x00007fc8f0d89660 in ?
 (version 23.9.2.47608 (official build))
2024.01.09 12:09:41.101831 [ 85 ] {} <Information> Application: shutting down
2024.01.09 12:09:41.101840 [ 85 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2024.01.09 12:09:41.102478 [ 87 ] {} <Trace> BaseDaemon: Received signal -2
2024.01.09 12:09:41.102511 [ 87 ] {} <Information> BaseDaemon: Stop SignalListener thread
2024.01.09 12:09:41.160849 [ 1 ] {} <Information> Application: Child process exited normally with code 70.

Expected behavior

Even if the bucket no longer exists, the server should still continue to run.