-
Notifications
You must be signed in to change notification settings - Fork 712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MyRocks: 1 hour stall in DelayWrite() during bulk load #348
Comments
This just happened again, about 20 minutes later, with identical stack traces |
Relevant config rocksdb rocksdb_max_open_files=-1 |
Can it perhaps be related to rocksdb_stats_dump_period_sec value? It is happening at regular intervals it seems. Mind you, MyRocks has 32100+ files open at this time (I opened a different issue about this) |
Could you paste "show global status like '%hash%';" output? We had RocksDB stall bugs at DelayWrite() before, but newer rocksdb revision fixed known bugs. |
Unfortunately my installation doesn't seem to have this info for some reason.... mysql> show global status like '%hash%'; Perhaps I can fetch that same info from a file in .git folder? |
The cmake process is supposed to assign values to MYSQL_GIT_HASH and ROCKSDB_GIT_HASH but that doesn't seem to have occurred in this case. Yes - send the git hash from your repository and from the rocksdb submodule. |
Dunno why, I followed instructions found on your wiki page here.... |
The first commits from this would be useful On Fri, Oct 21, 2016 at 11:06 AM, Rick Pizzi notifications@github.com
Mark Callaghan |
Mark: by �first commits", you mean more recent ones? Thanks RickRiccardo Pizzi Dogs have masters. Cats have staff. On 25/ott/2016, at 20:19, Mark Callaghan notifications@github.com wrote:
|
Please share the first (top) commit it shows. On Tue, Oct 25, 2016 at 11:23 AM, Rick Pizzi notifications@github.com
Mark Callaghan |
Here you go. pwd/home/rpizzi/mysql-fb git log | head -100commit 86587af
commit 8959b43
commit b0904cc
pwd/home/rpizzi/mysql-fb/rocksdb git log | head -100commit 6a17b07ca856e573fabd6345d70787d4e481f57b
commit 49d88be021bb83ddc4b8fe44dbf8f28fc00868b5
commit 0b63f51fbc4321bf67175be5c2b12f2884f23f90
commit 3981345be191dc676833341bac99372dc7940b86 Riccardo Pizzi Dogs have masters. Cats have staff. On 25/ott/2016, at 20:29, Mark Callaghan notifications@github.com wrote:
|
@RickPizzi We added new (much more efficient) data loading feature and fixed all known bugs. Could you please pull all changes (git pull --rebase), build, and try loading data again (See load optimizations at https://github.com/facebook/mysql-5.6/wiki/data-loading)? DelayWrite() stall is expected to disappear with our new data loading feature. |
Sure! Will do and report results back. Thank you!Riccardo Pizzi Dogs have masters. Cats have staff. On 31/ott/2016, at 22:55, Yoshinori Matsunobu notifications@github.com wrote:
|
Please re-open if you still the issue. |
Hi,
still trying to load our 830 tables stats server into MyRocks, we noticed a very long (> 1 hr) stall during the bulk load, when it was about to complete. Only 2 threads from myloader were left, but there has been no progress for more than one hour (SST files untouched, by looking at the files modification time).
I ran a pstack to check what was going on (output below). From what I can see, there were 2 threads trying to commit (do_bulk_commit) and a 3rd thread was a background thread trying to persist statistics (persist_stats). Out of the 2 committing thread, one was stuck in DelayWrite() and the other one waiting on a mutex.
After some more time everything magically resolved by itself (not sure if my pstack command was part of the black magic here).
Relevant output from pstack follows.
Thread 17 (Thread 0x7f9f735fe700 (LWP 30412)):
#0 0x00007fa102d4768c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fa1015aa37c in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /usr/local/lib64/libstdc++.so.6
#2 0x0000000000ca221c in rocksdb::WriteThread::BlockingAwaitState(rocksdb::WriteThread::Writer*, unsigned char) ()
#3 0x0000000000ca2348 in rocksdb::WriteThread::AwaitState(rocksdb::WriteThread::Writer_, unsigned char, rocksdb::WriteThread::AdaptationContext_) ()
#4 0x0000000000ca28a4 in rocksdb::WriteThread::JoinBatchGroup(rocksdb::WriteThread::Writer*) ()
#5 0x0000000000c0927f in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch_, rocksdb::WriteCallback_, unsigned long*, unsigned long, bool) ()
#6 0x0000000000c0b13b in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) ()
#7 0x0000000000baa4cd in myrocks::Rdb_dict_manager::commit(rocksdb::WriteBatch*, bool) ()
#8 0x0000000000badc75 in myrocks::Rdb_ddl_manager::persist_stats(bool) ()
#9 0x0000000000b8803c in myrocks::Rdb_background_thread::run() ()
#10 0x0000000000bdef54 in myrocks::Rdb_thread::thread_func(void*) ()
#11 0x00007fa102d43aa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fa100dacaad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fa0fc08f700 (LWP 6071)):
#0 0x00007fa100d70bdd in nanosleep () from /lib64/libc.so.6
#1 0x00007fa100da5d94 in usleep () from /lib64/libc.so.6
#2 0x0000000000be895f in rocksdb::DBImpl::DelayWrite(unsigned long) ()
#3 0x0000000000c0abfa in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch_, rocksdb::WriteCallback_, unsigned long*, unsigned long, bool) ()
#4 0x0000000000c0b13b in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) ()
#5 0x0000000000d37838 in rocksdb::TransactionImpl::Commit() ()
#6 0x0000000000b9ec6b in myrocks::Rdb_transaction_impl::commit_no_binlog() ()
#7 0x0000000000b8c25d in myrocks::ha_rocksdb::do_bulk_commit(myrocks::Rdb_transaction*) ()
#8 0x0000000000b92fac in myrocks::ha_rocksdb::update_write_row(unsigned char const_, unsigned char const_, bool) ()
#9 0x0000000000b9301d in myrocks::ha_rocksdb::write_row(unsigned char*) ()
#10 0x0000000000620334 in handler::ha_write_row(unsigned char*) ()
#11 0x0000000000e47f98 in ha_partition::write_row(unsigned char*) ()
#12 0x0000000000620334 in handler::ha_write_row(unsigned char*) ()
#13 0x000000000076daad in write_record(THD_, TABLE_, COPY_INFO_, COPY_INFO_) ()
#14 0x0000000000772f28 in mysql_insert(THD_, TABLE_LIST_, List&, List<List >&, List&, List&, enum_duplicates, bool) ()
#15 0x000000000078e420 in mysql_execute_command(THD_, unsigned long long_, unsigned long long*) ()
#16 0x0000000000793aef in mysql_parse(THD_, char_, unsigned int, Parser_state_, unsigned long long_, char_, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, char) ()
#17 0x0000000000795442 in dispatch_command(enum_server_command, THD, char_, unsigned int) ()
#18 0x0000000000753adc in do_handle_one_connection(THD*) ()
#19 0x0000000000753d89 in handle_one_connection ()
#20 0x00007fa102d43aa1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fa100dacaad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa0f42b5700 (LWP 6085)):
#0 0x00007fa102d4768c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fa1015aa37c in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /usr/local/lib64/libstdc++.so.6
#2 0x0000000000ca221c in rocksdb::WriteThread::BlockingAwaitState(rocksdb::WriteThread::Writer*, unsigned char) ()
#3 0x0000000000ca2348 in rocksdb::WriteThread::AwaitState(rocksdb::WriteThread::Writer_, unsigned char, rocksdb::WriteThread::AdaptationContext_) ()
#4 0x0000000000ca28a4 in rocksdb::WriteThread::JoinBatchGroup(rocksdb::WriteThread::Writer*) ()
#5 0x0000000000c0927f in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch_, rocksdb::WriteCallback_, unsigned long*, unsigned long, bool) ()
#6 0x0000000000c0b13b in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) ()
#7 0x0000000000d37838 in rocksdb::TransactionImpl::Commit() ()
#8 0x0000000000b9ec6b in myrocks::Rdb_transaction_impl::commit_no_binlog() ()
#9 0x0000000000b8c25d in myrocks::ha_rocksdb::do_bulk_commit(myrocks::Rdb_transaction*) ()
#10 0x0000000000b92fac in myrocks::ha_rocksdb::update_write_row(unsigned char const_, unsigned char const_, bool) ()
#11 0x0000000000b9301d in myrocks::ha_rocksdb::write_row(unsigned char*) ()
#12 0x0000000000620334 in handler::ha_write_row(unsigned char*) ()
#13 0x0000000000e47f98 in ha_partition::write_row(unsigned char*) ()
#14 0x0000000000620334 in handler::ha_write_row(unsigned char*) ()
#15 0x000000000076daad in write_record(THD_, TABLE_, COPY_INFO_, COPY_INFO_) ()
#16 0x0000000000772f28 in mysql_insert(THD_, TABLE_LIST_, List&, List<List >&, List&, List&, enum_duplicates, bool) ()
#17 0x000000000078e420 in mysql_execute_command(THD_, unsigned long long_, unsigned long long*) ()
#18 0x0000000000793aef in mysql_parse(THD_, char_, unsigned int, Parser_state_, unsigned long long_, char_, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, char) ()
#19 0x0000000000795442 in dispatch_command(enum_server_command, THD, char_, unsigned int) ()
#20 0x0000000000753adc in do_handle_one_connection(THD*) ()
#21 0x0000000000753d89 in handle_one_connection ()
#22 0x00007fa102d43aa1 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fa100dacaad in clone () from /lib64/libc.so.6
The text was updated successfully, but these errors were encountered: