Make a good overall story of replications, IO failure retries and tolerance levels

[grecinto]

Some errors are temporary, retry will isolate which ones are temp and which ones are not.
In the future, we may need to refine a story or feature of how to establish failover.

Perhaps track failure count and on threshold, failover to passive drive.
How does this work with rollback that also fails?
And how do we handle when rollback also fails? Data will be left in an inconsistent state. BUT hey, on such condition, what should we do?

Perhaps just logs error to alert the IT admin and let transactions to fail and fail over again, until the setup is fixed.
We need to think of the entire setup together with the EC based replication and the SOP replication active/passive. What is best story for handling IO errors, tolerance levels and what not.

[grecinto]

We are into this task now. :) A few failover candidates were picked up in previous task about replication & reinstating drives.

[grecinto]

Initial pass is accomplished. SOP in_red_fs has retry on each disk IO failure, raises failover on registry sector write and store repository write on disk that can't be rolled back.

Both of these scenarios causing auto failover seems the minimum use-cases for failover. More may be identified in the future, but these are a good start.