pthread unlock: Invalid argument during ErrorHandler::RecoverFromBGError

[benhannel]

It looks like this is the mutex in question: https://github.com/facebook/rocksdb/blob/master/db/error_handler.cc#L568

Does rocksdb have mechanisms to ensure that concurrently running cleanup operations complete before this mutex is destroyed?

Expected behavior

Rocksdb instances should not crash the process.

Actual behavior

pthread unlock: Invalid argument
*** Aborted at 1636596349 (Unix time, try 'date -d @1636596349') ***
*** Signal 6 (SIGABRT) (0x13) received by PID 19 (pthread TID 0x7f7301e55700) (linux TID 61682) (maybe from PID 19, UID 0) (code: -6), stack trace: ***
    @ 000000000028e68c folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /src/folly/folly/experimental/symbolizer/SignalHandler.cpp:457
    @ 0000000000000000 (unknown)
    @ 000000000004618b gsignal
    @ 0000000000025858 abort
    @ 0000000000c33ebf rocksdb::ErrorHandler::RecoverFromBGError(bool)
                       /src/rocksdb/db/error_handler.cc:0
    @ 0000000000d5931e rocksdb::SstFileManagerImpl::ClearError()
                       /src/rocksdb/file/sst_file_manager_impl.cc:308
    @ 0000000000000000 (unknown)
    @ 0000000000009608 start_thread
    @ 0000000000122292 clone

Steps to reproduce the behavior

I don't have a precise set of steps to reproduce, as this happened only once. However, I believe this happened during rocksdb shutdown after the machine had previously run out of disk space, which would explain the background error.

[benhannel]

cc @anand1976, who I believe wrote the auto-recovery code: a27fce4