[RFC] Disable automatic error recovery for user write failures

[anand1976]

In the current error recovery logic, background write errors during flush/compaction are automatically retried under some circumstances (NoSpace, retryable errors in distributed file systems). Normally, these errors are not visible to the user and we can try to recover from them in the background. However, if recovery takes a long time, the memtables eventually would become full with buffered writes and writes will be stopped by the write controller. When that happens, we currently return Status::Incomplete rather than indefinitely hang the write thread. There are 2 problems with this approach -

1. The Incomplete error may not be handled correctly by the user. It used to be returned only when the write_options.no_slowdown was set.
2. Other writes may be queued behind the incomplete write. If the background error recovery succeeds, the queued writes may be successful, which might cause inconsistency, especially with TransactionDB.

The solution is to stop all further writes once we return an error for a user write. This is accomplished in this PR as follows -

1. When the write controller stops writes and there is a background error, stop all further writes by setting the severity in bg_error_ to kHardError.
2. Return the bg_error_ rather than Status::Incomplete
3. Disable automatic error recovery in TransactionDB::Open() by setting db_options.max_bgerror_resume_count to 0. (Is this still required if we have Miss Spelling in README #1?)

[zhichao-cao]

How about compaction? Compaction do not do auto recovery and it just reschedule by itself. We set it to soft error. Should it be hard error?

[anand1976]

If the user has disabled auto recovery, I think we should set it to hard error even for compaction. Otherwise, too many pending compactions could also lead to a write stall.

[siying]

It's true that if users use RocksDB in certain way, as what MyRocks does, any write error might not be recoverable. How about other use cases, where writes are more or less independent, so one write failure can be skipped or independently retried later?

[anand1976]

That's a good point. Should we introduce an option to control this behavior? And perhaps it could apply to other user write failures, such as IO error during WAL append. The automatic recovery will flush memtables and create a new WAL if there was a WAL append failure. Any writes during that time will be failed, but subsequent writes will succeed.

[siying]

I think if we want to do that, we probably should go with an option. Do we have an API that allows users to manually recover? If we do, the option can be for manual recovery only.

[anand1976]

Yes, DB::Resume() allows users to manually recover. I added an option to freeze the DB on a user write failure. I kept it independent of auto recovery, since the auto recovery can continue as long as its confined to the background and not visible to the user. If the freeze options is set and there is a user visible failure (either due to reason kWriteCallback or write controller stoppage), then we put the DB in read-only mode and cancel any ongoing recovery.