Skip to content

[Bug] RocksDBDataStoreCommon::DropTable unable to drop table. #70

@MrGuin

Description

@MrGuin

DropTable fails during RocksdbCloud single longevity test. Since the first time the DropTable fails, it keeps retrying and failing for two days.
Detailed log file is on eloq@store-2:/mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/logs/node-6389/eloqkv.log.INFO.20250906-174059.2981528. The file is stored to eloq@eloq-super-server1/home/eloq/store_handler_issue_17/eloqkv.log.INFO.20250906-174059.2981528.

10838 [2025-09-06T17:54:05.118448 I 2981604]  txn: 4345906833 process acquire_all_intent_op_                                                                                                  10839 [2025-09-06T17:54:05.121887 I 2981604]  txn: 4345906833 process prepare_log_op_
 10840 [2025-09-06T17:54:05.133042 I 2981604]  txn: 4345906833 process post_all_intent_op_
 10841 [2025-09-06T17:54:05.135954 I 2981604]  txn: 4345906833 process acquire_all_lock_op_
 10842 [2025-09-06T17:54:05.136749 I 2981604]  txn: 4345906833 process commit_log_op_                                                                                                          10843 [2025-09-06T17:54:05.142589 I 2981604]  txn: 4345906833 process upsert_kv_table_op_
 10844 [2025-09-06T17:54:05.143138 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
 10845 [2025-09-06T17:54:05.144135 E 2981616]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
 10846 [2025-09-06T17:54:05.144574 W 2981678]  DataStoreHandler: Failed to do DropKvTable.
 10847 [2025-09-06T17:54:05.938580 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
 10848 [2025-09-06T17:54:05.939280 E 2981618]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
 10849 [2025-09-06T17:54:05.939636 W 2981678]  DataStoreHandler: Failed to do DropKvTable.
 10850 [2025-09-06T17:54:07.942220 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
 10851 [2025-09-06T17:54:07.943130 E 2981628]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached                                                                                                                10852 [2025-09-06T17:54:07.943652 W 2981678]  DataStoreHandler: Failed to do DropKvTable.
 10853 [2025-09-06T17:54:11.949412 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
 10854 [2025-09-06T17:54:11.950407 E 2981617]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached

465661 [2025-09-08T12:58:31.341657 W 2981678]  DataStoreHandler: Failed to do DropKvTable.
465662 [2025-09-08T12:58:33.276012 I 2981614]  Total memtables size: 0 MB
465663 [2025-09-08T12:58:33.276227 I 2981614]  Total SSTables size used in replay: 0 MB
465664 [2025-09-08T12:58:33.276286 I 2981614]  Total replay log size: 0 MB
465665 [2025-09-08T12:58:33.353834 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465666 [2025-09-08T12:58:33.354503 E 2981616]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
465667 [2025-09-08T12:58:33.354805 W 2981678]  DataStoreHandler: Failed to do DropKvTable.
465668 [2025-09-08T12:58:33.442333 I 2981713]  [Global dead lock detector]: gather lock dependancy information
465669 [2025-09-08T12:58:37.360330 I 2981678]  DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465670 [2025-09-08T12:58:37.361240 E 2981617]  Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data       /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions