Can't cleanup overlapped ranges due to out-or-order keys

[tabokie]

Bug Report

What version of TiKV are you using?

v5.2.2

What operating system and CPU are you using?

N/A

Steps to reproduce

Internal issue: https://internal.pingcap.net/jira/browse/ONCALL-4226

What did you expect?

What did happened?

failed to cleanup stale range: Keys must be added in order.

Workaround

Disable deletion by SST writer: raftstore.use-delete-range = true

[tabokie]

Suspect to be data corruption. The same TiKV node experienced out-of-order insertion during compaction later that week. We can't pursue the investigation because user already scale-in the problematic node.

Next time, it's advised to run the following commands to check whether the corruption is at memory level (block-cache) or persistent level:

for filename in /path/to/tikv/data/*.sst; do
  echo dump $filename && tikv-ctl ldb dump --path=$filename > /dev/null
done