Fix The range specified is invalid for the current size of the resource when reading from azure disk with plain_rewritable and encryption#86400
Conversation
e122bc5 to
def2687
Compare
| auto normalized_path = normalizeDirectoryPath(path); | ||
| if (normalized_path.empty()) | ||
| { | ||
| LOG_TRACE(getLogger("MetadataStorageFromPlainObjectStorageTransaction"), "Skipping creation of a directory with an empty path"); |
There was a problem hiding this comment.
Should we throw an exception instead?
And can we print out the original path, so we know if something wrong with normalizeDirectoryPath?
There was a problem hiding this comment.
I essentially moved the check to src/Disks/DiskEncrypted.cpp
There was a problem hiding this comment.
Should we throw an exception instead?
I essentially moved the check to src/Disks/DiskEncrypted.cpp
Reverting that, because it has some undesired side effects, e.g.:
tests/queries/0_stateless/03362_merge_tree_with_background_refresh.sh
[ip-172-31-8-86] 2025.08.30 03:18:34.873339 [ 1335461 ] {56d32d72-96e6-48eb-b4dd-7a0f6fd04002} <Error> executeQuery: Code: 281. DB::Exception: Directory path '' is empty. (DICTIONARY_IS_EMPTY) (version 25.9.1.1) (from [::ffff:127.0.0.1]:61872) (query 1, line 2) (in query: CREATE TABLE writer (`s` String) ORDER BY tuple() SETTINGS table_disk = true, disk = disk(name = `03362_writer_test`, type = object_storage, object_storage_type = '[HIDDEN]', metadata_type = '[HIDDEN]', path = '[HIDDEN]')), Stack trace (when copying this message, always include the lines below):
0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000001f7bce32
1. ./build.release/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fad999e
2. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000904a38e
3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009049f80
4. DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a11162b
5. ./build.release/./src/Disks/ObjectStorages/MetadataStorageFromPlainObjectStorage.cpp:297: DB::MetadataStorageFromPlainObjectStorageTransaction::createDirectory(String const&) @ 0x0000000015052943
6. ./build.release/./src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:708: void std::__function::__policy_invoker<void (std::shared_ptr<DB::IMetadataTransaction>)>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::DiskObjectStorageTransaction::createDirectories(String const&)::$_0, void (std::shared_ptr<DB::IMetadataTransaction>)>>(std::__function::__policy_storage const*, std::shared_ptr<DB::IMetadataTransaction>&&) @ 0x0000000015018de7
7. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000015018c2f
8. ./build.release/./src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:1067: DB::DiskObjectStorageTransaction::commit(std::variant<std::monostate, DB::MetaInKeeperCommitOptions<zkutil::ZooKeeper>, DB::MetaInKeeperCommitOptions<DB::ZooKeeperWithFaultInjection>> const&) @ 0x0000000015017e8f
9. ./build.release/./src/Disks/ObjectStorages/DiskObjectStorage.cpp:389: DB::DiskObjectStorage::createDirectories(String const&) @ 0x0000000015005775
10. ./build.release/./src/Storages/MergeTree/MergeTreeData.cpp:409: DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x000000001903f500
11. ./build.release/./src/Storages/StorageMergeTree.cpp:170: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x0000000019000a9c
12. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:41: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:ne190107]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, 0>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001941a5f6
13. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:851: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001941720d
14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001897a6fb
15. ./build.release/./src/Interpreters/InterpreterCreateQuery.cpp:1951: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLG26. start_thread @ 0x0000000000094ac3
27. __GI___clone3 @ 0x0000000000126850
(version 25.9.1.1)
Leaving it as a no-op.
| auto read_buf = object_storage->readObject(object, settings); | ||
| readStringUntilEOF(local_path, *read_buf); | ||
| if (metadata->size_bytes == 0) | ||
| LOG_TRACE(log, "The object with the key '{}' has size 0, skipping the read", remote_metadata_path); |
There was a problem hiding this comment.
If we skipping the read, should we return here?
There was a problem hiding this comment.
In theory (though unlikely), top-level objects can be created for an empty directory. No return ensures that they are read and added to the in-memory map.
|
|
This reverts commit a970bd5.
|
There is something wrong with the upgrade check |
|
tests/config/install.sh changes should not impact; merging the master to see if it reproduces |
|
It looks like the issue this PR addresses was reproduced for the 25.8 release, because of the new tests/config/config.d/storage_conf_03602.xml config, which is why the upgrade checks have failed. |
|
I'll add the config in a separate PR after the fix is merged to 25.8 |
52d46b3
Cherry pick #86400 to 25.8: Fix `The range specified is invalid for the current size of the resource` when reading from azure disk with plain_rewritable and encryption
…current size of the resource` when reading from azure disk with plain_rewritable and encryption
Backport #86400 to 25.8: Fix `The range specified is invalid for the current size of the resource` when reading from azure disk with plain_rewritable and encryption
Cherry pick #86400 to 25.6: Fix `The range specified is invalid for the current size of the resource` when reading from azure disk with plain_rewritable and encryption
…current size of the resource` when reading from azure disk with plain_rewritable and encryption
Backport #86400 to 25.6: Fix `The range specified is invalid for the current size of the resource` when reading from azure disk with plain_rewritable and encryption
Do not write empty blobs to Azure blob storage, which happens when creating an encrypted disk with an empty
path.When the disk is reloaded (e.g., because of pod restart), reading from an empty blob fails with
416 The range specified is invalid for the current size of the resource. The range specified is invalid for the current size of the resource:https://pastila.nl/?000448db/8f3c31d2c5a6c3dffd32af6d6cbc547b#yLMbaEXWJBYP6pz3CmHKrg==
Additionally, skip reading from empty blobs for objects that already exist.
The issue did not reproduce with Azurite, so I added a test that uses
openbucketforpublicciAzure bucket.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fix
416 The range specified is invalid for the current size of the resource. The range specified is invalid for the current size of the resourcewhen reading empty blobs from Azure blob storage for plain_rewritable disk.Documentation entry for user-facing changes