Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Small improvements for tables on write-once (s3_plain)/read-only (web) disks #59170

Merged
merged 16 commits into from Jan 29, 2024

Commits on Jan 26, 2024

  1. tests: add s3_plain disk

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    e56728a View commit details
    Browse the repository at this point in the history
  2. Do not drop parts for static (ro/write-once) storages for Replicated …

    …engine
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    6751e01 View commit details
    Browse the repository at this point in the history
  3. Add a note for MetadataStorageFromPlainObjectStorage to support trans…

    …actions
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    41ebaef View commit details
    Browse the repository at this point in the history
  4. Disable mutable operations into {Replicated,}MergeTree with static st…

    …orage
    
    Mutable operations includes:
    - INSERT
    - OPTIMIZE
    - parts manipulation
    - ...
    
    But does not include DROP, since you need an ability to DROP it.
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    f3b4f28 View commit details
    Browse the repository at this point in the history
  5. Implement MetadataStorageFromPlainObjectStorageTransaction::removeDir…

    …ectory (for DROP TABLE)
    
    This is required to remove format_version.txt for MergeTree after DROP
    
        (lldb) bt
        * thread 192, name = 'BgSchPool', stop reason = breakpoint 1.1
            frame 0: 0x0000000016c7f452 clickhouse`__cxa_throw(thrown_object=0x00007ffd7a53c480, tinfo=0x00000000011602a0, dest=(clickhouse`DB::Exception::~Exception() at Exception.h:40)) + 18 at cxa_exception.cpp:258
            frame 1: 0x00000000109bf709 clickhouse`DB::IMetadataTransaction::throwNotImplemented() + 137 at IMetadataStorage.h:151
            frame 2: 0x00000000109bf126 clickhouse`DB::IMetadataTransaction::removeDirectory(this=<unavailable>, (null)=<unavailable>) + 6 at IMetadataStorage.h:96
            frame 3: 0x0000000010968978 clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::removeMetadataRecursive(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1, path_to_remove="tmp_insert_all_2_2_0") + 376 at DiskObjectStorageTransaction.cpp:364
            frame 4: 0x000000001096893d clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::removeMetadataRecursive(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1, path_to_remove="store/27f/27f6f9d2-ab2e-4817-b771-0e706e485b15/") + 317 at DiskObjectStorageTransaction.cpp:362
            frame 5: 0x0000000010965a7f clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::execute(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1) + 95 at DiskObjectStorageTransaction.cpp:372
            frame 6: 0x0000000010963a2f clickhouse`DB::DiskObjectStorageTransaction::commit(this=0x00007ffea22b6168) + 79 at DiskObjectStorageTransaction.cpp:903
            frame 7: 0x00000000109577e5 clickhouse`DB::DiskObjectStorage::removeSharedRecursive(this=<unavailable>, path="store/27f/27f6f9d2-ab2e-4817-b771-0e706e485b15/", keep_all_batch_data=false, file_names_remove_metadata_only=size=0) + 133 at DiskObjectStorage.cpp:426
            frame 8: 0x000000001095af79 clickhouse`DB::DiskObjectStorage::removeRecursive(this=<unavailable>, path=<unavailable>) + 57 at DiskObjectStorage.h:79
          * frame 9: 0x0000000010b5c3d5 clickhouse`DB::DatabaseCatalog::dropTableFinally(this=0x00007ffff7096800, table=0x00007ffe853e1bd0) + 1365 at DatabaseCatalog.cpp:1317
            frame 10: 0x0000000010b5b7e7 clickhouse`DB::DatabaseCatalog::dropTableDataTask(this=0x00007ffff7096800) + 2375 at DatabaseCatalog.cpp:1270
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    9ce18bb View commit details
    Browse the repository at this point in the history
  6. Fix MetadataStorageFromPlainObjectStorage::iterateDirectory() (add pr…

    …efix)
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    d79c1f7 View commit details
    Browse the repository at this point in the history
  7. tests: cover DROP TABLE on s3_plain disk

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    046a7a5 View commit details
    Browse the repository at this point in the history
  8. Prohibit DROP from static storages

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    1ee723d View commit details
    Browse the repository at this point in the history
  9. Prohibit CREATE on static storages (ATTACH should be used instead)

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    aafc149 View commit details
    Browse the repository at this point in the history
  10. tests: rewrite test for MergeTree engines in s3_plain using ATTACH

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    d99fb4c View commit details
    Browse the repository at this point in the history
  11. Revert "Prohibit DROP from static storages"

    It is actually OK to DROP at least from read-only disk, as for the
    write-once it should be OK as well, since CREATE is prohibited anyway,
    so there should be no leftovers
    
    CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/stress_test__asan_.html
    CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/integration_tests__asan__[3_4].html
    
    This reverts commit 082e627.
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    66a40bd View commit details
    Browse the repository at this point in the history
  12. tests/test_disk_over_web_server: update expected assertion for static…

    … storage
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    7cc7fb8 View commit details
    Browse the repository at this point in the history
  13. Do not create format_version.txt on write-once storages

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    7ba31bf View commit details
    Browse the repository at this point in the history
  14. Revert "Prohibit CREATE on static storages (ATTACH should be used ins…

    …tead)"
    
    CREATE is used by Replicated database even for ATTACH, and anyway ATTACH
    creates format_version.txt as well (without previuos patch).
    
    This reverts commit 323f1f320ebe7e588d443abf78efa2c60193e7a9.
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    90ab986 View commit details
    Browse the repository at this point in the history
  15. tests: adjust 02980_s3_plain_DROP_TABLE tests

    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 26, 2024
    Copy the full SHA
    332924a View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2024

  1. tests: fix 02980_s3_plain_DROP_TABLE_ReplicatedMergeTree flakiness

    In case of commit failed (likely due to fault injection in tests) part
    name will have different name, not all_0_0_0, so replace numbers with X.
    
    Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
    azat committed Jan 27, 2024
    Copy the full SHA
    71cd1ef View commit details
    Browse the repository at this point in the history