In-memory engine: wrong-total in jepsen test

[SpadeA-Tang]

Bug Report

What version of TiKV are you using?

Master

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

Jepsen pass

What did happened?

{:latency-graph {:valid? true},
  :rate-graph {:valid? true},
  :valid? true},
 :workload
 {:SI
  {:valid? false,
   :read-count 142404,
   :error-count 1,
   :first-error
   {:type :wrong-total,
    :total 61,
    :op
    {:type :ok,
     :f :read,
     :process 1,
     :time 38284047629,
     :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
     :txn-info {:start_ts 449754326792667154},
     :index 33719}},
   :errors
   {:wrong-total
    {:count 1,
     :first
     {:type :wrong-total,
      :total 61,
      :op
      {:type :ok,
       :f :read,
       :process 1,
       :time 38284047629,
       :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
       :txn-info {:start_ts 449754326792667154},
       :index 33719}},
     :worst
     {:type :wrong-total,
      :total 61,
      :op
      {:type :ok,
       :f :read,
       :process 1,
       :time 38284047629,
       :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
       :txn-info {:start_ts 449754326792667154},
       :index 33719}},
     :last
     {:type :wrong-total,
      :total 61,
      :op
      {:type :ok,
       :f :read,
       :process 1,
       :time 38284047629,
       :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
       :txn-info {:start_ts 449754326792667154},
       :index 33719}},
     :lowest
     {:type :wrong-total,
      :total 61,
      :op
      {:type :ok,
       :f :read,
       :process 1,
       :time 38284047629,
       :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
       :txn-info {:start_ts 449754326792667154},
       :index 33719}},
     :highest
     {:type :wrong-total,
      :total 61,
      :op
      {:type :ok,
       :f :read,
       :process 1,
       :time 38284047629,
       :value {0 12, 1 25, 2 10, 3 7, 6 5, 7 2},
       :txn-info {:start_ts 449754326792667154},
       :index 33719}}}}},
  :plot {:valid? true},
  :valid? false},
 :valid? false}

[SpadeA-Tang]

I have found some bugs or missing part leading to this error:

1. Delete lock directly violates the write atomicity. We can make the write complete then delete the lock.
2. Missing handles in remove peers, merge, apply snapshot: all theses cases should evict the relevant regions.

[SpadeA-Tang]

In addition, using same seqno for the whole batch also leads to this type of error. If there's a put lock and delete lock in the same batch with the same user key, the delete will be hidden by the put.
We use increment the seqno for each key just like what does in rocksdb.