Many disks are shutdown due to heavy workload

[polyrabbit]

What happened:
During a heavy IO workload, we observed more than 280 pending write requests to the cache store, which overwhelmed the disk and caused slow IO. The bad disk dictator thus shuts it down forever even it recovers shortly. But worse, we lost the read cache forever.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
  1.2-beta1

[jiefenghuang]

#4749
In this PR, we have added a fault tolerance range to reduce misjudgments, which aligns with your scenario. If needed, maxIOErrors can be adjusted as configurable.

[polyrabbit]

Thanks, but we already included this commit in our build:

[jiefenghuang]

In this writing scenario, maybe turn on maxStageWrite is better

[polyrabbit]

When one disk cache store is shutdown. Will we lose staged files in that store?

[jiefenghuang]

files still exist.

[jiefenghuang]

Conclusion: There are two scenarios to consider. One is long-term high-load writing, for which writing directly to object storage is recommended (resource issue: disk-cache or object storage bandwidth). The other is short-term high-load writing, for which the concurrency limit can be lifted to redirect and distribute writes to the object storage. This issue will be temporarily closed for now.

[davies]

We should enable max-stage-write for this issue, maybe 10 as the default value.