titan_stress failure

[yiwu-arbug]

[root@b57112560f33 /]# /usr/local/titandb/bin/titandb_stress --ops_per_thread=10000 --threads=40 --value_size_mult=33 --set_options_one_in=1000 --compact_range_one_in=1000 --acquire_snapshot_one_in=1 --ingest_external_file_one_in=1000 --compact_files_one_in=0 --checkpoint_one_in=0 --backup_one_in=0 --nooverwritepercent=0 --delrangepercent=10  --delpercent=15 --readpercent=15 --writepercent=30 --iterpercent=15 --prefixpercent=15 --max_background_compactions=20 --max_background_flushes=20 --enable_pipelined_write=true --min_blob_size=64
2019/09/03-08:42:52  Initializing db_stress
RocksDB version           : 5.18
Format version            : 2
TransactionDB             : false
Column families           : 10
Clear CFs one in          : 1000000
Number of threads         : 40
Ops per thread            : 10000
Time to live(sec)         : unused
Read percentage           : 15%
Prefix percentage         : 15%
Write percentage          : 30%
Delete percentage         : 15%
Delete range percentage   : 10%
No overwrite percentage   : 0%
Iterate percentage        : 15%
DB-write-buffer-size      : 0
Write-buffer-size         : 67108864
Iterations                : 10
Max key                   : 1048576
Ratio #ops/#keys          : 0.381470
Num times DB reopens      : 10
Batches/snapshots         : 0
Do update in place        : 0
Num keys per lock         : 4
Compression               : Snappy
Checksum type             : kCRC32c
Max subcompactions        : 1
Memtablerep               : prefix_hash
Test kill odd             : 0
------------------------------------------------
DB path: [/tmp/rocksdbtest-0/dbstress]
Choosing random keys with no overwrite
Creating 2621440 locks
2019/09/03-08:42:52  Initializing worker threads
2019/09/03-08:42:52  Starting database operations
2019/09/03-08:42:54  Reopening database for the 1th time
DB path: [/tmp/rocksdbtest-0/dbstress]
2019/09/03-08:42:56  Reopening database for the 2th time
DB path: [/tmp/rocksdbtest-0/dbstress]
[CF 2] Dropping and recreating column family. new name: 10
delete error: Operation aborted: Column family drop
put or merge error: Operation aborted: Column family drop
terminate called recursively
put or merge error: Operation aborted: Column family drop
terminate called without an active exception
terminate called recursively
Aborted (core dumped)

[CF 2] Dropping and recreating column family. new name: 10
put or merge error: Operation aborted: Column family drop
terminate called without an active exception

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff1ffff700 (LWP 123692)]
0x00007ffff6b76277 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install libatomic-4.8.5-36.el7_6.2.x86_64 scylla-libgcc73-7.3.1-1.2.el7.centos.x86_64 scylla-libstdc++73-7.3.1-1.2.el7.centos.x86_64 snappy-1.1.0-3.el7.x86_64
(gdb) bt
#0  0x00007ffff6b76277 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6b77968 in __GI_abort () at abort.c:90
#2  0x00007ffff74b8575 in __gnu_cxx::__verbose_terminate_handler() () from /opt/scylladb/lib64/libstdc++.so.6
#3  0x00007ffff74b6166 in ?? () from /opt/scylladb/lib64/libstdc++.so.6
#4  0x00007ffff74b61b1 in std::terminate() () from /opt/scylladb/lib64/libstdc++.so.6
#5  0x0000000000a797f6 in rocksdb::NonBatchedOpsStressTest::TestPut(rocksdb::(anonymous namespace)::ThreadState*, rocksdb::WriteOptions&, rocksdb::ReadOptions const&, std::vector<int, std::allocator<int> > const&, std::vector<long, std::allocator<long> > const&, char (&) [1048576], std::unique_ptr<rocksdb::MutexLock, std::default_delete<rocksdb::MutexLock> >&) ()
#6  0x0000000000a76aeb in rocksdb::StressTest::OperateDb(rocksdb::(anonymous namespace)::ThreadState*) ()
#7  0x0000000000a8d406 in rocksdb::StressTest::ThreadBody(void*) ()
#8  0x0000000000dbfae4 in rocksdb::(anonymous namespace)::StartThreadWrapper(void*) ()
#9  0x00007ffff77b8e25 in start_thread (arg=0x7fff1ffff700) at pthread_create.c:308
#10 0x00007ffff6c3ebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

[yiwu-arbug]

reported by @tabokie

[yiwu-arbug]

Reproduced locally. It crashes pretty quick. I got one crash with the above stack trace, and another one crashing at https://github.com/pingcap/titan/blob/468ddc9727428bf85403a63e6fa3f245d81e6de7/src/blob_format.cc#L171 where state_ equals to kNormal. Still debugging.

[yiwu-arbug]

The failure at TestPut is caused by Titan hitting background error "Column family drop", which is caused by a CF dropped while GC on the CF is still running. Filed #71 to track.

[yiwu-arbug]

Hit another db_stress failure, which is a db_stress bug and fixed in upstream. The thing is I didn't update titandb_stress to the version in rocksdb 6.4.x. Will make the update.

[yiwu-arbug]

The file state transit error seems to be caused by file size property collector not being called during flush and compaction sometimes. Still investigating.

[yiwu-arbug]

titandb_stress pass locally after patching with #72, #76 and #78.