run integration-tests with tikv v6.5.8 failed

[bxq2011hust]

run integration-tests on tikv 6.5.8 without api-version = 2, got error

failures:

---- txn_batch_mutate_pessimistic stdout ----
thread 'txn_batch_mutate_pessimistic' panicked at tests/common/mod.rs:37:10:
called `Result::unwrap()` on an `Err` value: ExtractedErrors([StringError("Multiple key errors: [KvError { message: \"Api version in request does not match with TiKV storage, cmd: raw_delete_range, storage: V1, request: V2\" }]")])
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

how to set the client to api-version = 1?
use TransactionClient::new should use v1, but get the same error

[pingyu]

Currently integration tests are hard coded to v2.

Set clients to use API v1 by TransactionClient::new() or RawClient::new().

To address the error, you may also need to use a v1 client in clear_tikv().

[bxq2011hust]

Currently integration tests are hard coded to v2.

Set clients to use API v1 by TransactionClient::new() or RawClient::new().

To address the error, you may also need to use a v1 client in clear_tikv().

I notice that and modify the TransactionClient::new_with_config to use Config::default() which will set Keyspace::Disable still get the same error. I guess somewhere(raw_delete_range) not properly handle this API version

I'm not familiar with tikv-rust-client, can you please check it? @pingyu

let client =
        TransactionClient::new_with_config(pd_addrs(), Config::default())

[pingyu]

I meant the error was raised from clear_tikv().

Change the Config::default().with_default_keyspace() in the following two lines to Config::default().

    let raw_client =
        RawClient::new_with_config(pd_addrs(), Config::default().with_default_keyspace())
...
...
    let txn_client =
        TransactionClient::new_with_config(pd_addrs(), Config::default().with_default_keyspace())

[bxq2011hust]

Thank you for your reply, after modifying clear_tikv(), the test cause TiKV v6.5.8 crash with errors😢
Do you have any suggestions？

tikv quit: exit status 1
[2024/03/26 09:37:04.176 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 5"] [term=5] [raft_id=87] [region_id=86]
[2024/03/26 09:37:04.176 +08:00] [INFO] [raft.rs:1151] ["became candidate at term 6"] [term=6] [raft_id=87] [region_id=86]
[2024/03/26 09:37:04.176 +08:00] [INFO] [raft.rs:1235] ["became leader at term 6"] [term=6] [raft_id=87] [region_id=86]
[2024/03/26 09:37:04.176 +08:00] [INFO] [peer.rs:5622] ["require updating max ts"] [initial_status=25769803862] [region_id=86]
[2024/03/26 09:37:04.176 +08:00] [INFO] [endpoint.rs:699] ["register observe region"] [region="id: 86 start_key: 7480000000000000FF4C00000000000000F8 end_key: 7480000000000000FF4E00000000000000F8 region_epoch { conf_ver: 1 version: 43 } peers { id: 87 store_id: 1 }"]
[2024/03/26 09:37:04.176 +08:00] [INFO] [endpoint.rs:357] ["Resolver initialized"] [pending_data_index=0] [snapshot_index=70] [observe_id=ObserveId(83)] [region=10]
[2024/03/26 09:37:04.176 +08:00] [INFO] [pd.rs:1715] ["succeed to update max timestamp"] [region_id=86]
[2024/03/26 09:37:04.176 +08:00] [INFO] [endpoint.rs:357] ["Resolver initialized"] [pending_data_index=0] [snapshot_index=6] [observe_id=ObserveId(84)] [region=86]
[2024/03/26 09:37:09.040 +08:00] [INFO] [client.rs:848] ["set cluster version to 6.5.8-dirty"]
[2024/03/26 09:37:14.970 +08:00] [FATAL] [lib.rs:509] ["assertion failed: !req.get_start_key().is_empty()"] [backtrace="   0: backtrace::capture::Backtrace::new\n   1: tikv_util::set_panic_hook::{{closure}}\n   2: std::panicking::rust_panic_with_hook\n   3: std::panicking::begin_panic_handler::{{closure}}\n   4: std::sys_common::backtrace::__rust_end_short_backtrace\n   5: _rust_begin_unwind\n   6: core::panicking::panic_fmt\n   7: core::panicking::panic\n   8: <grpcio::server::Handler<F> as grpcio::server::CloneableHandler>::handle\n   9: grpcio::call::server::execute\n  10: grpcio::env::poll_queue\n  11: std::sys_common::backtrace::__rust_begin_short_backtrace\n  12: core::ops::function::FnOnce::call_once{{vtable.shim}}\n  13: std::sys::unix::thread::Thread::new::thread_start\n  14: __pthread_joiner_wake\n"] [location=/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tikv/src/server/service/kv.rs:521] [thread_name=grpc-server-3]

[pingyu]

Emmm... txn_client.unsafe_destroy_range(..) is not allowed in API v1.

Just comment out the the two lines. It's not necessary in API v1.

    let txn_client =
        TransactionClient::new_with_config(pd_addrs(), Config::default().with_default_keyspace())
            .await
            .unwrap();
    txn_client.unsafe_destroy_range(..).await.unwrap();

[bxq2011hust]

Thanks, comment out works. I ran the test many times and the following three tests have not passed, the txn_get_for_update sometimes not passed, is this expected？ @pingyu

successes:
    txn_bank_transfer
    txn_batch_mutate_optimistic
    txn_batch_mutate_pessimistic
    txn_get_for_update
    txn_get_timestamp
    txn_insert_duplicate_keys
    txn_key_exists
    txn_lock_keys
    txn_lock_keys_error_handle
    txn_pessimistic
    txn_pessimistic_delete
    txn_pessimistic_heartbeat
    txn_pessimistic_rollback
    txn_scan
    txn_scan_reverse
    txn_split_batch
    txn_unsafe_destroy_range
    txn_update_safepoint

failures:

---- txn_crud stdout ----
init finish with 43 regions
thread 'txn_crud' panicked at tests/integration_tests.rs:74:5:
assertion failed: txn.get("foo".to_owned()).await?.is_none()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- txn_read stdout ----
init finish with 43 regions
Error: MultipleKeyErrors([KeyError(KeyError { locked: None, retryable: "", abort: "Error(Txn(Error(Mvcc(Error(DefaultNotFound { key: [109, 68, 66, 58, 49, 0, 0, 0, 255, 0, 251, 0, 0, 0, 0, 0, 0, 255, 0, 104, 84, 97, 98, 108, 101, 58, 255, 49, 48, 255, 0, 0, 0, 0, 0, 255, 0, 0, 0, 247, 0, 0, 0, 0, 251, 249, 198, 29, 179, 240, 215, 255, 232] })))))", conflict: None, already_exist: None, deadlock: None, commit_ts_expired: None, txn_not_found: None, commit_ts_too_large: None, assertion_failed: None, primary_mismatch: None })])

---- txn_scan_reverse_multi_regions stdout ----
init finish with 43 regions
Error: MultipleKeyErrors([KeyError(KeyError { locked: None, retryable: "", abort: "Error(Txn(Error(Mvcc(Error(DefaultNotFound { key: [109, 68, 68, 76, 74, 111, 98, 72, 255, 105, 255, 115, 116, 111, 114, 121, 0, 255, 0, 0, 252, 0, 0, 0, 0, 0, 255, 0, 0, 104, 0, 0, 0, 0, 0, 255, 0, 0, 79, 255, 0, 0, 0, 0, 255, 0, 0, 0, 0, 247, 0, 0, 0, 252, 249, 198, 29, 179, 186, 39, 255, 250] })))))", conflict: None, already_exist: None, deadlock: None, commit_ts_expired: None, txn_not_found: None, commit_ts_too_large: None, assertion_failed: None, primary_mismatch: None })])


failures:
    txn_crud
    txn_read
    txn_scan_reverse_multi_regions

[pingyu]

is this expected?

No. There seems to be bugs.