Tracking issue of TiKV disk full

[tier-cap]

Development Task

This issue describes a new feature that makes TiKV can avoid panicking from disk full failure, and provide space reclamation support at the same time. The spec:

• TiKV shouldn't panic even if its disk usage is almost full
• Even if most TiKV instances in a cluster are almost full, the cluster should be still readable.
• Even if most TiKV instances in a cluste are almost full, users should still can release space by drop table or truncate table.

Task list

• TiKV won't panic for disk full #10597
• Support drop/truncate table when TiKV disk full pingcap/tidb#27665
• Support setting of whether txn exec is allowed when TiKV disk full. client-go#286
• Support operation advise from leader to follower when TiKV disk full pingcap/kvproto#803

Bug List

• One bug about the CommitMerge when TiKV disk full #10960
• One bug about the user level disk full error code #10959
• Another read failed problem during to region merging when TiKV disk full #10926
• One bug about handling disk usage status reported by the followers #10925
• One bug on whether cur cmd is allowed or not when TiKV is disk full #10895
• One bug about the chosing majority set when handling peer diskusage report #10894
• Expansion fails when TiKV diskusage is already full #10868
• The data distribution is abnormal when the disk is full and expansion #10840
• The leader distribution is abnormal when the disk is full and expansion  #10839
• Read failed when tikv is disk full #10819
• Expansion fails when TiKV diskusage is almost full #10801

Left & Low Priority

• Running log need archive #11010
• TiKV should stop the sst import when disk usage is very high #10665
• Space management of RocksDB compaction operation #11109
• PD scheduling strategy under abnormal TiKV conditions #11110
• RocksDB compaction needs to take into account the storage space left. #11232
• Snap mgr should be banlanced with disk space usage #11236
• Should the disk full error code returned by TiKV be unified with MySQL 1021? pingcap/tidb#29557

[tisonkun]

Hi @tier-cap ! Thanks for creating this tracking issue. You may create issue for each subtask dedicatedly, see also pingcap/tidb#26022 . The rationale of tracing issues instead of pull requests can be found here.

For example, you may create an issue that gets closed once #10562 resolved.

[tisonkun]

We are tracking issue by issues, why do you update the description redundant to show PRs? And let the PRs refer to this one instead of subtask so that the subtask remains even if the PR merged.

[DanielZhangQD]

For some users that only have small disks, e.g. 10GB disk, it's really necessary to have the features in this issue to improve the disk usage ratio and the stability of the TiDB clusters.