db_stress hangs/segfaults for a small test case

[yhr]

Expected behavior

That db_stress exist gracefully for the command line

Actual behavior

A deadlock(?) at the test completion (after "Verification successful") for debug builds
..and with a segfault for non-debug builds:

2022/02/01-12:55:08  Verification successful
pthread lock: Invalid argument
Aborted

Reverting commit f07c569 fixes the issue.

Steps to reproduce the behavior

./db_stress --ops_per_thread=1000 --reopen=5

[ltamasi]

Thanks for reporting @yhr ! I've run your repro under TSAN and indeed there are some races, for instance this one which I believe is the immediate cause of the crash:

WARNING: ThreadSanitizer: data race (pid=3862971)
  Write of size 1 at 0x7ffc46e12400 by main thread:
    #0 pthread_mutex_destroy /home/engshare/third-party2/gcc/9.x/src/gcc-10.x/libsanitizer/tsan/tsan_interceptors.cc:1200 (libtsan.so.0+0x39dde)
    #1 rocksdb::port::Mutex::~Mutex() port/port_posix.cc:79 (db_stress+0x95c60d)
    #2 rocksdb::SharedState::~SharedState() db_stress_tool/db_stress_shared_state.h:175 (db_stress+0x4f4105)
    #3 rocksdb::RunStressTest(rocksdb::StressTest*) db_stress_tool/db_stress_driver.cc:62 (db_stress+0x4f4105)
    #4 rocksdb::db_stress_tool(int, char**) db_stress_tool/db_stress_tool.cc:345 (db_stress+0x4b8dfb)
    #5 main db_stress_tool/db_stress.cc:21 (db_stress+0x478da1)

  Previous atomic read of size 1 at 0x7ffc46e12400 by thread T52:
    #0 pthread_mutex_lock /home/engshare/third-party2/gcc/9.x/src/gcc-10.x/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4069 (libtsan.so.0+0x50ca0)
    #1 rocksdb::port::Mutex::Lock() port/port_posix.cc:82 (db_stress+0x95c64a)
    #2 rocksdb::MutexLock::MutexLock(rocksdb::port::Mutex*) util/mutexlock.h:32 (db_stress+0x4f0ee0)
    #3 rocksdb::DbVerificationThread(void*) db_stress_tool/db_stress_common.cc:135 (db_stress+0x4f0ee0)
    #4 StartThreadWrapper env/env_posix.cc:447 (db_stress+0x84346f)

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x00000001d400)

  Thread T52 (tid=3863213, finished) created by main thread at:
    #0 pthread_create /home/engshare/third-party2/gcc/9.x/src/gcc-10.x/libsanitizer/tsan/tsan_interceptors.cc:964 (libtsan.so.0+0x613c3)
    #1 StartThread env/env_posix.cc:457 (db_stress+0x845102)
    #2 rocksdb::CompositeEnvWrapper::StartThread(void (*)(void*), void*) env/composite_env_wrapper.h:318 (db_stress+0x81d2e0)
    #3 rocksdb::EnvWrapper::StartThread(void (*)(void*), void*) include/rocksdb/env.h:1520 (db_stress+0x4bb2e0)
    #4 rocksdb::RunStressTest(rocksdb::StressTest*) db_stress_tool/db_stress_driver.cc:99 (db_stress+0x4f35cd)
    #5 rocksdb::db_stress_tool(int, char**) db_stress_tool/db_stress_tool.cc:345 (db_stress+0x4b8dfb)
    #6 main db_stress_tool/db_stress.cc:21 (db_stress+0x478da1)