mysqld crashed when running a RQG transactions test, not reproducible at will

[mariadb-DanielLee]

Build tested:

RocksDB:
myrocks@rocksdb:~/builds/mytest/mysql-5.6/rocksdb$ git show
commit c8e01ef
Author: Satnam Singh satnamsingh@fb.com
Date: Thu Nov 5 13:30:51 2015 -0800

MyRocks:
myrocks@rocksdb:~/builds/mytest/mysql-5.6$ git show
commit f03213b
Author: Yoshinori Matsunobu yoshinori@fb.com
Date: Fri Dec 4 13:42:46 2015 -0800

RQG command:
perl gentest.pl --dsn=dbi:mysql:host=localhost:port=3306:user=root:database=test:mysql_socket=/tmp/mysql.sock --gendata=conf/transactions/transactions.zz --grammar=conf/transactions/transactions.yy --threads=10 --queries=1000

From error log:

2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33822)
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33823)
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33824)
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33825)
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33826)
2015-12-10 08:37:38 37291 [Note] LibRocksDB:Adding new file 7127 range ('000540A480000E64' @ 279877784: 1, '000540F88000091C' @ 279861409: 1) to level 2 but overlapping with existing file 7159 '000540870080000DBE' @ 0: 1 '000540A480000E64' @ 0: 1
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33827)
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33828)
mysqld: /home/myrocks/builds/mytest/mysql-5.6/rocksdb/db/version_set.cc:1283: void rocksdb::VersionStorageInfo::AddFile(int, rocksdb::FileMetaData*, rocksdb::Logger*): Assertion `false' failed.
2015-12-10 08:37:38 37291 [Note] RocksDB: Resume filtering dropped index (0,33829)


Back trace:myrocks@rocksdb:~/builds/reptest/data1/install.db$ gdb ../../mysql-5.6/sql/mysqld core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../../mysql-5.6/sql/mysqld...done.
[New LWP 30600]
[New LWP 31678]
[New LWP 30510]
[New LWP 31846]
[New LWP 30507]
[New LWP 30512]
[New LWP 30597]
[New LWP 30509]
[New LWP 30549]
[New LWP 31908]
[New LWP 30511]
[New LWP 31929]
[New LWP 31681]
[New LWP 31866]
[New LWP 31887]
[New LWP 31826]
[New LWP 31680]
[New LWP 31677]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./mysqld --defaults-file=/home/myrocks/builds/reptest/reptestMaster.cnf --rocks'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f853d188cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
Traceback (most recent call last):
 File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
   from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0  0x00007f853d188cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f853d18c0d8 in __GI_abort () at abort.c:89
#2  0x00007f853d181b86 in __assert_fail_base (fmt=0x7f853d2d2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
   assertion=assertion@entry=0xf07645 "false", 
   file=file@entry=0xfed2a0 "/home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_set.cc", line=line@entry=1283, 
   function=function@entry=0xfef520 <rocksdb::VersionStorageInfo::AddFile(int, rocksdb::FileMetaData*, rocksdb::Logger*)::__PRETTY_FUNCTION__> "void rocksdb::VersionStorageInfo::AddFile(int, rocksdb::FileMetaData*, rocksdb::Logger*)")
   at assert.c:92
#3  0x00007f853d181c32 in __GI___assert_fail (assertion=0xf07645 "false", 
   file=0xfed2a0 "/home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_set.cc", line=1283, 
   function=0xfef520 <rocksdb::VersionStorageInfo::AddFile(int, rocksdb::FileMetaData*, rocksdb::Logger*)::__PRETTY_FUNCTION__> "void rocksdb::VersionStorageInfo::AddFile(int, rocksdb::FileMetaData*, rocksdb::Logger*)") at assert.c:101
#4  0x0000000000d0cbd8 in rocksdb::VersionStorageInfo::AddFile (this=this@entry=0x7f84f00555e0, level=level@entry=2, 
   f=0x7f84f006d470, info_log=info_log@entry=0x2bf8718)
   at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_set.cc:1283
#5  0x0000000000de20dc in MaybeAddFile (this=<optimized out>, this=<optimized out>, f=<optimized out>, level=2, 
   vstorage=0x7f84f00555e0) at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_builder.cc:343
#6  SaveTo (vstorage=<optimized out>, this=<optimized out>)
   at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_builder.cc:283
#7  rocksdb::VersionBuilder::SaveTo (this=this@entry=0x7f84f00277e0, vstorage=vstorage@entry=0x7f84f00555e0)
   at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_builder.cc:363
#8  0x0000000000d147ff in rocksdb::VersionSet::LogAndApply (this=0x2c1baa0, column_family_data=0x2c1f220, 
   mutable_cf_options=..., edit=edit@entry=0x7f84f0085670, mu=mu@entry=0x2c11290, db_directory=0x2c25d10, 
   new_descriptor_log=new_descriptor_log@entry=false, new_cf_options=new_cf_options@entry=0x0)
   at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/version_set.cc:2072
#9  0x0000000000dc33d6 in rocksdb::CompactionJob::InstallCompactionResults (this=this@entry=0x7f8536614bc0, 
   mutable_cf_options=..., db_mutex=db_mutex@entry=0x2c11290)
   at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/compaction_job.cc:890
#10 0x0000000000dc681c in rocksdb::CompactionJob::Install (this=this@entry=0x7f8536614bc0, mutable_cf_options=..., 
   db_mutex=db_mutex@entry=0x2c11290) at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/compaction_job.cc:523
#11 0x0000000000cb60d5 in rocksdb::DBImpl::BackgroundCompaction (this=this@entry=0x2c11070, 
   made_progress=made_progress@entry=0x7f8536615120, job_context=job_context@entry=0x7f8536615160, 
   log_buffer=log_buffer@entry=0x7f8536615320) at /home/myrocks/builds/reptest/mysql-5.6/rocksdb/db/db_impl.cc:2887
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) quit

[hermanlee]

Could you send me the terminal server login for us to get the core and binary?

[yoshinorim]

Siying says it seems a known issue in RocksDB and they're working on fixing.

[mariadb-DanielLee]

Unfortunately, the core file was removed and no longer available when I refreshed the database.

Regards,
—
Daniel Lee, QA Engineer
MariaDB, www.mariadb.com

On Dec 10, 2015, at 11:58 AM, Herman Lee notifications@github.com wrote:

Could you send me the terminal server login for us to get the core and binary?

—
Reply to this email directly or view it on GitHub #123 (comment).

[yoshinorim]

Here is a reproducible test case in RocksDB. https://reviews.facebook.net/D51783
Reid is working on fixing it in RocksDB.