Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault: current_page_acq_t::current_page_for_read #2222

Closed
coffeemug opened this issue Apr 7, 2014 · 17 comments
Closed

Segfault: current_page_acq_t::current_page_for_read #2222

coffeemug opened this issue Apr 7, 2014 · 17 comments
Assignees
@srh
Labels
pr:high tp:bug
Milestone
1.12.2

Comments

@coffeemug
Copy link
Contributor

Stack trace from the user:

error: Error in src/arch/runtime/thread_pool.cc at line 341:
error: Segmentation fault from reading the address 0x10.
error: Backtrace:
addr2line: 'rethinkdb': No such file
error: Thu Apr  3 18:52:09 2014

       1: format_backtrace(bool) at 0x10e5a93 (rethinkdb)
       2: report_fatal_error(char const*, int, char const*, ...) at 0x112bf94 (rethinkdb)
       3: linux_thread_pool_t::sigsegv_handler(int, siginfo*, void*) at 0xb633a4 (rethinkdb)
       4: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f8b49fdfcb0] at 0x7f8b49fdfcb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
       5: alt::current_page_acq_t::current_page_for_read(cache_account_t*) at 0x10cc3ad (rethinkdb)
       6: buf_read_t::get_data_read(unsigned int*) at 0x10c83fa (rethinkdb)
       7: blob::expose_tree_from_block_ids(buf_parent_t, access_t, int, long, long, temporary_acq_tree_node_t*, blob::touches_end_t, buffer_group_t*, blob_acq_t*) at 0x10dcfdb (rethinkdb)
       8: blob::expose_tree_from_block_ids(buf_parent_t, access_t, int, long, long, temporary_acq_tree_node_t*, blob::touches_end_t, buffer_group_t*, blob_acq_t*) at 0x10dcf74 (rethinkdb)
       9: blob_t::expose_region(buf_parent_t, access_t, long, long, buffer_group_t*, blob_acq_t*) at 0x10dd4c4 (rethinkdb)
       10: blob_t::expose_all(buf_parent_t, access_t, buffer_group_t*, blob_acq_t*) at 0x10dd76d (rethinkdb)
       11: get_data(rdb_value_t const*, buf_parent_t) at 0xc095c6 (rethinkdb)
       12: lazy_json_t::get() const at 0xc09784 (rethinkdb)
       13: rget_cb_t::handle_pair(scoped_key_value_t&&, concurrent_traversal_fifo_enforcer_signal_t) at 0xc93fd9 (rethinkdb)
       14: concurrent_traversal_adapter_t::handle_pair_coro(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t) at 0xd11917 (rethinkdb)
       15: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (concurrent_traversal_adapter_t::*)(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> (concurrent_traversal_adapter_t*, scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> >::run_action() at 0xd11818 (rethinkdb)
       16: coro_t::run() at 0xb67f1e (rethinkdb)
error: Exiting.
Trace/breakpoint trap (core dumped)

Running on:

3.8.0-37-generic #53~precise1-Ubuntu SMP Wed Feb 19 21:37:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

@rlshepherd -- I have a few more questions:

  • What is the version of RethinkDB? (you can get it by running rethinkdb --version)
  • What exactly is the script doing during the crash?
@coffeemug coffeemug added this to the 1.12.x milestone Apr 7, 2014
@coffeemug
Copy link
Contributor Author

Pinging @srh (and possibly @danielmewes)

@rlshepherd
Copy link 

$ rethinkdb --version
rethinkdb 1.12.1-0ubuntu1~precise (GCC 4.6.3)

The script is appending to an array for all documents matching the filter (in this case, a field containing a URL). I'm using the python API:

r.db('mydb').table('mytable').filter({'url': url}).update({
    'shares': r.row['shares'].append({
      'count': NewShares(url, platform),
      'timestamp': r.now(),
      'platform': platform
    })
}).run()

The script calling this code ran every 5 minutes for ~3 days, so the array should be about 800+ items when the crashing started occurring. ~200 documents in the table.

I also get this warning when starting the rethinkdb server, don't know if it is related:

Could not turn off filesystem caching for database file: ...

@coffeemug
Copy link
Contributor Author

Thank you! Would it be possible for you to zip up the files and send them to us? (You can send them to slava@rethinkdb.com if they fit into an e-mail; if not we'll set up a share for you).

Could not turn off filesystem caching for database file

Are the RethinkDB data files on an encrypted/compressed file system? In this case Rethink can't use direct IO, so it switches to use the page cache. (I opened an issue in the doc repository to write a troubleshooting entry for it -- rethinkdb/docs#259)

@rlshepherd
Copy link

Will follow up on email, thank you!

@coffeemug
Copy link
Contributor Author

For core devs looking at this issue, the data files, core dump, and the script that produces the segfault are in newton:/home/coffeemug/issue_2222/.

@danielmewes
Copy link
Member

The assembler code of the crash point in current_page_acq_t::current_page_for_read() looks like this (the line with the => is where the crash happens):

...
   0x00000000010cc390 <+48>:    callq  0xb20f40 <_ZNK8signal_t19wait_lazily_orderedEv>
   0x00000000010cc395 <+53>:    mov    %rbp,%rdi
   0x00000000010cc398 <+56>:    callq  0x10dfea0 <_ZNK3alt22timestamped_page_ptr_t3hasEv>
   0x00000000010cc39d <+61>:    test   %al,%al
   0x00000000010cc39f <+63>:    jne    0x10cc410 <_ZN3alt18current_page_acq_t21current_page_for_readEP15cache_account_t+176>
   0x00000000010cc3a1 <+65>:    mov    0x10(%rbx),%r13
   0x00000000010cc3a5 <+69>:    mov    0x28(%rbx),%r12
   0x00000000010cc3a9 <+73>:    mov    0x30(%rbx),%rbx
=> 0x00000000010cc3ad <+77>:    cmpb   $0x0,0x10(%rbx)
   0x00000000010cc3b1 <+81>:    jne    0x10cc3f0 <_ZN3alt18current_page_acq_t21current_page_for_readEP15cache_account_t+144>
   0x00000000010cc3b3 <+83>:    cmpq   $0x0,0x8(%rbx)
   0x00000000010cc3b8 <+88>:    lea    0x8(%rbx),%r15
   0x00000000010cc3bc <+92>:    je     0x10cc440 <_ZN3alt18current_page_acq_t21current_page_for_readEP15cache_account_t+224>
   0x00000000010cc3c2 <+98>:    mov    %r15,%rdi
   0x00000000010cc3c5 <+101>:   mov    0x8(%rsp),%rbx
   0x00000000010cc3ca <+106>:   mov    0x10(%rsp),%rbp
   0x00000000010cc3cf <+111>:   mov    0x18(%rsp),%r12
   0x00000000010cc3d4 <+116>:   mov    0x20(%rsp),%r13
   0x00000000010cc3d9 <+121>:   mov    0x28(%rsp),%r14
   0x00000000010cc3de <+126>:   mov    0x30(%rsp),%r15
   0x00000000010cc3e3 <+131>:   add    $0x38,%rsp
   0x00000000010cc3e7 <+135>:   jmpq   0x10dfdf0 <_ZNK3alt10page_ptr_t17get_page_for_readEv>
...
   0x00000000010cc410 <+176>:   mov    %rbp,%rdi
   0x00000000010cc413 <+179>:   mov    0x8(%rsp),%rbx
   0x00000000010cc418 <+184>:   mov    0x10(%rsp),%rbp
   0x00000000010cc41d <+189>:   mov    0x18(%rsp),%r12
   0x00000000010cc422 <+194>:   mov    0x20(%rsp),%r13
   0x00000000010cc427 <+199>:   mov    0x28(%rsp),%r14
   0x00000000010cc42c <+204>:   mov    0x30(%rsp),%r15
   0x00000000010cc431 <+209>:   add    $0x38,%rsp
   0x00000000010cc435 <+213>:   jmpq   0x10dfed0 <_ZNK3alt22timestamped_page_ptr_t17get_page_for_readEv>

For comparison the source code of that function with asserts removed (those don't get compiled in release mode):

page_t *current_page_acq_t::current_page_for_read(cache_account_t *account) {
    read_cond_.wait();
    if (snapshotted_page_.has()) {
        return snapshotted_page_.get_page_for_read();
    }
    return current_page_->the_page_for_read(help(), account);
}

One can see the call to <_ZNK8signal_t19wait_lazily_orderedEv> in the assembly, as well as a call to <_ZNK3alt22timestamped_page_ptr_t3hasEv>. It seems that if the snapshotted_page_.has() call returns, the program continues at <+176> of the disassembly, where the call to <_ZNK3alt22timestamped_page_ptr_t17get_page_for_readEv> is visible.
In this case however, snapshotted_page_.has() was apparently false.
Now it seems that the call to current_page_->the_page_for_read(help(), account) has been inlined here, so I'm not sure where this fails exactly. My suspicion would be that current_page_ is NULL somehow.

@AtnNn do we have a full symbol file for the builds somewhere?

@danielmewes
Copy link
Member

Oh actually the comparison that crashes is probably the line
guarantee(!is_deleted_); in current_page_t::the_page_for_read().
In cmpb $0x0,0x10(%rbx), cmpb compares a byte (which makes sense for bools), the $0x0 would be the constant false value, and I imagine that 0x10 is the offset of is_deleted_ in current_page_t. %rbx is probably supposed to hold the pointer to the current_page_t.
The crash message said Segmentation fault from reading the address 0x10., which would make sense if %rbx was 0, i.e. current_page_ was NULL.

@danielmewes
Copy link
Member

Now the question is how both current_page_ and snapshotted_page_ can be NULL and empty respectively, because that should not happen.
I'll see if I can find a problem with the code, but @srh should definitely take a look too.

@danielmewes
Copy link
Member

I think we can get into this situation if a deleted page gets snapshotted. Then the snapshotted_page_ will be empty (because page_ptr_t::has() checks whether the pointer to the page_t is NULL, which it is in case of a deleted page), and we also will not have a current_page_, because we are snapshotted.

Now generally nobody should attempt to read a deleted page. I will try a bit more to reproduce this problem locally. So far I didn't have any luck.

@danielmewes
Copy link
Member

Hah, got a crash. It was necessary to have a concurrent read transaction, for example r.table().filter(function(x) {return true;}).

My backtrace is different, but I believe it's the same underlying problem.

error: Error in src/rdb_protocol/lazy_json.cc at line 19:
error: Guarantee failed: [res == archive_result_t::SUCCESS] Deserialization of rdb value failed with error archive_result_t::RANGE_ERROR.
error: Backtrace:
error: Mon Apr  7 14:51:46 2014

       1: rethinkdb_backtrace(void**, int) at backtrace.cc:229
       2: backtrace_t::backtrace_t() at 0x2bac55b (../rethinkdb/build/debug/rethinkdb)
       3: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at main.cc:24
       4: format_backtrace(bool) at stl_tree.h:139
       5: report_fatal_error(char const*, int, char const*, ...) at backtrace.cc:154
       6: get_data(rdb_value_t const*, buf_parent_t) at op.cc:142
       7: lazy_json_t::get() const at op.cc:146
       8: rget_cb_t::handle_pair(scoped_key_value_t&&, concurrent_traversal_fifo_enforcer_signal_t) at scoped.hpp:260
       9: concurrent_traversal_adapter_t::handle_pair_coro(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t) at depth_first_traversal.cc:84
       10: std::_Mem_fn<void (concurrent_traversal_adapter_t::*)(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)>::operator()(concurrent_traversal_adapter_t*, scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t) const at counted.hpp:141
       11: _ZNSt5_BindIFSt7_Mem_fnIM30concurrent_traversal_adapter_tFvP18scoped_key_value_tP15semaphore_acq_t27fifo_enforcer_write_token_tN14auto_drainer_t6lock_tEEEPS1_S3_S5_S6_S8_EE6__callIvJEJLi0ELi1ELi2ELi3ELi4EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x10f at keys.hpp:118
       12: _ZNSt5_BindIFSt7_Mem_fnIM30concurrent_traversal_adapter_tFvP18scoped_key_value_tP15semaphore_acq_t27fifo_enforcer_write_token_tN14auto_drainer_t6lock_tEEEPS1_S3_S5_S6_S8_EEclIJEvEET0_DpOT_+0x26 at counted.hpp:98
       13: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (concurrent_traversal_adapter_t::*)(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> (concurrent_traversal_adapter_t*, scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> >::run_action() at counted.hpp:140
       14: callable_action_wrapper_t::run() at thread_pool.cc:384
       15: coro_t::run() at stl_algobase.h:624
       16: void coro_t::spawn_now_dangerously<std::_Bind<std::_Mem_fn<void (concurrent_traversal_adapter_t::*)(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> (concurrent_traversal_adapter_t*, scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> >(std::_Bind<std::_Mem_fn<void (concurrent_traversal_adapter_t::*)(scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> (concurrent_traversal_adapter_t*, scoped_key_value_t*, semaphore_acq_t*, fifo_enforcer_write_token_t, auto_drainer_t::lock_t)> const&) at depth_first_traversal.cc:65
       17: concurrent_traversal_adapter_t::handle_pair(scoped_key_value_t&&) at depth_first_traversal.cc:26
       18: btree_depth_first_traversal(counted_t<counted_buf_lock_t>, key_range_t const&, depth_first_traversal_callback_t*, direction_t) at stl_iterator.h:892
       19: btree_depth_first_traversal(counted_t<counted_buf_lock_t>, key_range_t const&, depth_first_traversal_callback_t*, direction_t) at vector.tcc:357
       20: btree_depth_first_traversal(superblock_t*, key_range_t const&, depth_first_traversal_callback_t*, direction_t) at 0x1ee0f8f (../rethinkdb/build/debug/rethinkdb)
       21: btree_concurrent_traversal(superblock_t*, key_range_t const&, concurrent_traversal_callback_t*, direction_t) at semaphore.hpp:111
       22: rdb_rget_slice(btree_slice_t*, key_range_t const&, superblock_t*, ql::env_t*, ql::batchspec_t const&, std::vector<boost::variant<ql::map_wire_func_t, ql::group_wire_func_t, ql::filter_wire_func_t, ql::concatmap_wire_func_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::allocator<boost::variant<ql::map_wire_func_t, ql::group_wire_func_t, ql::filter_wire_func_t, ql::concatmap_wire_func_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > const&, boost::optional<boost::variant<ql::count_wire_func_t, ql::sum_wire_func_t, ql::avg_wire_func_t, ql::min_wire_func_t, ql::max_wire_func_t, ql::reduce_wire_func_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > const&, ql::sorting_t, rdb_protocol_t::rget_read_response_t*) at operations.tcc:104
       23: rdb_read_visitor_t::operator()(rdb_protocol_t::rget_read_t const&) at protocol.hpp:659
       24: void boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::internal_visit<rdb_protocol_t::rget_read_t const>(rdb_protocol_t::rget_read_t const&, int) at visitation_impl.hpp:140
       25: boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<rdb_read_visitor_t>, void const*, rdb_protocol_t::rget_read_t>(int, boost::detail::variant::invoke_visitor<rdb_read_visitor_t>&, void const*, rdb_protocol_t::rget_read_t*, mpl_::bool_<false>) at protocol.hpp:670
       26: boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<rdb_read_visitor_t>, void const*, rdb_protocol_t::rget_read_t, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<rdb_read_visitor_t>&, void const*, rdb_protocol_t::rget_read_t*, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_, int) at variant.hpp:848
       27: boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::result_type boost::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<5l>, rdb_protocol_t::point_read_t, boost::mpl::l_item<mpl_::long_<4l>, rdb_protocol_t::rget_read_t, boost::mpl::l_item<mpl_::long_<3l>, rdb_protocol_t::distribution_read_t, boost::mpl::l_item<mpl_::long_<2l>, rdb_protocol_t::sindex_list_t, boost::mpl::l_item<mpl_::long_<1l>, rdb_protocol_t::sindex_status_t, boost::mpl::l_end> > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke_visitor<rdb_read_visitor_t>, void const*, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_>(int, int, boost::detail::variant::invoke_visitor<rdb_read_visitor_t>&, void const*, mpl_::bool_<false>, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<5l>, rdb_protocol_t::point_read_t, boost::mpl::l_item<mpl_::long_<4l>, rdb_protocol_t::rget_read_t, boost::mpl::l_item<mpl_::long_<3l>, rdb_protocol_t::distribution_read_t, boost::mpl::l_item<mpl_::long_<2l>, rdb_protocol_t::sindex_list_t, boost::mpl::l_item<mpl_::long_<1l>, rdb_protocol_t::sindex_status_t, boost::mpl::l_end> > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*) at visitation_impl.hpp:127
       28: boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::result_type boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::internal_apply_visitor_impl<boost::detail::variant::invoke_visitor<rdb_read_visitor_t>, void const*>(int, int, boost::detail::variant::invoke_visitor<rdb_read_visitor_t>&, void const*) at visitation_impl.hpp:127
       29: boost::detail::variant::invoke_visitor<rdb_read_visitor_t>::result_type boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::internal_apply_visitor<boost::detail::variant::invoke_visitor<rdb_read_visitor_t> >(boost::detail::variant::invoke_visitor<rdb_read_visitor_t>&) const at visitation_impl.hpp:205
       30: rdb_read_visitor_t::result_type boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::apply_visitor<rdb_read_visitor_t>(rdb_read_visitor_t&) const at 0x297d635 (../rethinkdb/build/debug/rethinkdb)
       31: rdb_read_visitor_t::result_type boost::apply_visitor<rdb_read_visitor_t, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const>(rdb_read_visitor_t&, boost::variant<rdb_protocol_t::point_read_t, rdb_protocol_t::rget_read_t, rdb_protocol_t::distribution_read_t, rdb_protocol_t::sindex_list_t, rdb_protocol_t::sindex_status_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) at boost_types.hpp:131
       32: rdb_protocol_t::store_t::protocol_read(rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, btree_slice_t*, superblock_t*, signal_t*) at functional:1449
       33: btree_store_t<rdb_protocol_t>::read(metainfo_checker_t<rdb_protocol_t> const&, rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, order_token_t, read_token_pair_t*, signal_t*) at btree_store.cc:499
       34: store_subview_t<rdb_protocol_t>::read(metainfo_checker_t<rdb_protocol_t> const&, rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, order_token_t, read_token_pair_t*, signal_t*) at functional:1215
       35: store_subview_t<rdb_protocol_t>::read(metainfo_checker_t<rdb_protocol_t> const&, rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, order_token_t, read_token_pair_t*, signal_t*) at functional:1215
       36: listener_t<rdb_protocol_t>::perform_read(rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t) at scoped.hpp:96
       37: boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>::operator()(listener_t<rdb_protocol_t>*, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t) const at debug.hpp:133
       38: void boost::_bi::list7<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::_bi::value<rdb_protocol_t::read_t>, boost::_bi::value<state_timestamp_t>, boost::_bi::value<order_token_t>, boost::_bi::value<fifo_enforcer_read_token_t>, boost::_bi::value<mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::value<auto_drainer_t::lock_t> >::operator()<boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>&, boost::_bi::list0&, int) at new_allocator.h:108
       39: boost::_bi::bind_t<void, boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>, boost::_bi::list7<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::_bi::value<rdb_protocol_t::read_t>, boost::_bi::value<state_timestamp_t>, boost::_bi::value<order_token_t>, boost::_bi::value<fifo_enforcer_read_token_t>, boost::_bi::value<mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::value<auto_drainer_t::lock_t> > >::operator()() at base.hpp:1410
       40: callable_action_instance_t<boost::_bi::bind_t<void, boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>, boost::_bi::list7<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::_bi::value<rdb_protocol_t::read_t>, boost::_bi::value<state_timestamp_t>, boost::_bi::value<order_token_t>, boost::_bi::value<fifo_enforcer_read_token_t>, boost::_bi::value<mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::value<auto_drainer_t::lock_t> > > >::run_action() at space_optimized.hpp:1260
       41: callable_action_wrapper_t::run() at thread_pool.cc:384
       42: coro_t::run() at stl_algobase.h:624
       43: coro_t* coro_t::spawn_sometime<boost::_bi::bind_t<void, boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>, boost::_bi::list7<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::_bi::value<rdb_protocol_t::read_t>, boost::_bi::value<state_timestamp_t>, boost::_bi::value<order_token_t>, boost::_bi::value<fifo_enforcer_read_token_t>, boost::_bi::value<mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::value<auto_drainer_t::lock_t> > > >(boost::_bi::bind_t<void, boost::_mfi::mf6<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>, auto_drainer_t::lock_t>, boost::_bi::list7<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::_bi::value<rdb_protocol_t::read_t>, boost::_bi::value<state_timestamp_t>, boost::_bi::value<order_token_t>, boost::_bi::value<fifo_enforcer_read_token_t>, boost::_bi::value<mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::value<auto_drainer_t::lock_t> > > const&) at storage.hpp:129
       44: listener_t<rdb_protocol_t>::on_read(rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>) at storage.hpp:178
       45: boost::_mfi::mf5<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >::operator()(listener_t<rdb_protocol_t>*, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>) const at space_optimized.hpp:1273
       46: void boost::_bi::list6<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::arg<5> >::operator()<boost::_mfi::mf5<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::list5<rdb_protocol_t::read_t&, state_timestamp_t&, order_token_t&, fifo_enforcer_read_token_t&, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>&> >(boost::_bi::type<void>, boost::_mfi::mf5<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >&, boost::_bi::list5<rdb_protocol_t::read_t&, state_timestamp_t&, order_token_t&, fifo_enforcer_read_token_t&, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>&>&, int) at disk_backed_queue_wrapper.hpp:27
       47: void boost::_bi::bind_t<void, boost::_mfi::mf5<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::list6<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::arg<5> > >::operator()<rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >(rdb_protocol_t::read_t&, state_timestamp_t&, order_token_t&, fifo_enforcer_read_token_t&, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>&) at scoped.hpp:29
       48: boost::detail::function::void_function_obj_invoker5<boost::_bi::bind_t<void, boost::_mfi::mf5<void, listener_t<rdb_protocol_t>, rdb_protocol_t::read_t const&, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >, boost::_bi::list6<boost::_bi::value<listener_t<rdb_protocol_t>*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::arg<5> > >, void, rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >::invoke(boost::detail::function::function_buffer&, rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>) at disk_backed_queue_wrapper.hpp:41
       49: boost::function5<void, rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >::operator()(rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>) const at mem_fn_template.hpp:619
       50: mailbox_t<void (rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>)>::read_impl_t::read(read_stream_t*) at bind.hpp:596
       51: mailbox_manager_t::mailbox_read_coroutine(peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long) at vclock.cc:40
       52: std::_Mem_fn<void (mailbox_manager_t::*)(peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)>::operator()(mailbox_manager_t*, peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long) const at semilattice_manager.tcc:429
       53: _ZNSt5_BindIFSt7_Mem_fnIM17mailbox_manager_tFv9peer_id_t11threadnum_tmPSt6vectorIcSaIcEElEEPS1_S2_S3_mS7_lEE6__callIvJEJLi0ELi1ELi2ELi3ELi4ELi5EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x13b at semilattice_manager.tcc:423
       54: _ZNSt5_BindIFSt7_Mem_fnIM17mailbox_manager_tFv9peer_id_t11threadnum_tmPSt6vectorIcSaIcEElEEPS1_S2_S3_mS7_lEEclIJEvEET0_DpOT_+0x26 at semilattice_manager.tcc:416
       55: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (mailbox_manager_t::*)(peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> (mailbox_manager_t*, peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> >::run_action() at view.hpp:14
       56: callable_action_wrapper_t::run() at thread_pool.cc:384
       57: coro_t::run() at stl_algobase.h:624
       58: void coro_t::spawn_now_dangerously<std::_Bind<std::_Mem_fn<void (mailbox_manager_t::*)(peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> (mailbox_manager_t*, peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> >(std::_Bind<std::_Mem_fn<void (mailbox_manager_t::*)(peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> (mailbox_manager_t*, peer_id_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long)> const&) at semilattice_manager.hpp:26
       59: mailbox_manager_t::on_message(peer_id_t, read_stream_t*) at vclock.cc:26
       60: message_multiplexer_t::run_t::on_message(peer_id_t, read_stream_t*) at bind.hpp:864
       61: connectivity_cluster_t::send_message(peer_id_t, send_message_write_callback_t*) at 0x2a5594e (../rethinkdb/build/debug/rethinkdb)
       62: non-virtual thunk to connectivity_cluster_t::send_message(peer_id_t, send_message_write_callback_t*) at cluster.hpp:266
       63: message_multiplexer_t::client_t::send_message(peer_id_t, send_message_write_callback_t*) at read_manager.tcc:214
       64: send(mailbox_manager_t*, raw_mailbox_t::address_t, mailbox_write_callback_t*) at scoped.hpp:144
       65: void send<rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> >(mailbox_manager_t*, mailbox_t<void (rdb_protocol_t::read_t, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, mailbox_addr_t<void (rdb_protocol_t::read_response_t)>)>::address_t, rdb_protocol_t::read_t const&, state_timestamp_t const&, order_token_t const&, fifo_enforcer_read_token_t const&, mailbox_addr_t<void (rdb_protocol_t::read_response_t)> const&) at variant.hpp:1574
       66: void listener_read<rdb_protocol_t>(mailbox_manager_t*, listener_business_card_t<rdb_protocol_t>::read_mailbox_t::address_t const&, rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, state_timestamp_t, order_token_t, fifo_enforcer_read_token_t, signal_t*) at stl_tree.h:1054
       67: broadcaster_t<rdb_protocol_t>::single_read(rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, fifo_enforcer_sink_t::exit_read_t*, order_token_t, signal_t*) at stl_map.h:467
       68: broadcaster_t<rdb_protocol_t>::read(rdb_protocol_t::read_t const&, rdb_protocol_t::read_response_t*, fifo_enforcer_sink_t::exit_read_t*, order_token_t, signal_t*) at broadcaster.cc:234
       69: master_t<rdb_protocol_t>::client_t::perform_request(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, signal_t*) at variant.hpp:1823
       70: multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::perform_request(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t) at visitation_impl.hpp:279
       71: std::_Mem_fn<void (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::*)(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t)>::operator()(multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t*, boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t) const at variant.hpp:651
       72: _ZNSt5_BindIFSt7_Mem_fnIMN25multi_throttling_server_tIN5boost7variantIN22master_business_card_tI14rdb_protocol_tE14read_request_tENS6_15write_request_tENS2_6detail7variant5void_ESB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_EENS6_28inner_client_business_card_tEP8master_tIS5_ENSF_8client_tEE8client_tEFvRKSC_N14auto_drainer_t6lock_tEEEPSJ_SC_SN_EE6__callIvJEJLi0ELi1ELi2EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x9d at variant.hpp:677
       73: _ZNSt5_BindIFSt7_Mem_fnIMN25multi_throttling_server_tIN5boost7variantIN22master_business_card_tI14rdb_protocol_tE14read_request_tENS6_15write_request_tENS2_6detail7variant5void_ESB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_SB_EENS6_28inner_client_business_card_tEP8master_tIS5_ENSF_8client_tEE8client_tEFvRKSC_N14auto_drainer_t6lock_tEEEPSJ_SC_SN_EEclIJEvEET0_DpOT_+0x26 at backup_holder.hpp:37
       74: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::*)(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t)> (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t*, boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, auto_drainer_t::lock_t)> >::run_action() at variant.hpp:659
       75: callable_action_wrapper_t::run() at thread_pool.cc:384
       76: coro_t::run() at stl_algobase.h:624
       77: coro_t* coro_t::spawn_sometime<std::_Bind<std::_Mem_fn<void (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::*)(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t)> (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t*, boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, auto_drainer_t::lock_t)> >(std::_Bind<std::_Mem_fn<void (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::*)(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, auto_drainer_t::lock_t)> (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t*, boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, auto_drainer_t::lock_t)> const&) at visitation_impl.hpp:256
       78: multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::on_request(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) at function_template.hpp:566
       79: std::_Mem_fn<void (multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t::*)(boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&)>::operator()(multi_throttling_server_t<boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, master_business_card_t<rdb_protocol_t>::inner_client_business_card_t, master_t<rdb_protocol_t>*, master_t<rdb_protocol_t>::client_t>::client_t*, boost::variant<master_business_card_t<rdb_protocol_t>::read_request_t, master_business_card_t<rdb_protocol_t>::write_request_t, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const at visitation_impl.hpp:256

Looks like a problem with snapshotting.

@danielmewes danielmewes modified the milestones: 1.12.2, 1.12.x Apr 7, 2014
@danielmewes danielmewes self-assigned this Apr 7, 2014
@srh srh assigned srh and unassigned danielmewes Apr 8, 2014
@srh
Copy link
Contributor

srh commented Apr 8, 2014

We are assigning it to me, because it'll require dealing with some awful blob code.

@srh
Copy link
Contributor

srh commented Apr 8, 2014

Working on this in sam_2222 (off v1.12.x of course).

@srh
Copy link
Contributor

srh commented Apr 8, 2014

This might be (should be) fixed in branch sam_2222, I'm doing some testing.

@srh
Copy link
Contributor

srh commented Apr 8, 2014

We can confirm that this doesn't crash in the way v1.12.x crashes on the data for this issue with the scripts that @danielmewes created.

@srh
Copy link
Contributor

srh commented Apr 8, 2014

This is fixed and merged to v1.12.x as of cea6c39.

@srh srh closed this as completed Apr 8, 2014
@AtnNn
Copy link
Member

AtnNn commented Apr 9, 2014

@rlshepherd We have just released RethinkDB 1.12.2, which includes a fix for this issue.

@rlshepherd
Copy link

Thank you very much!

On Thu, Apr 10, 2014 at 5:06 AM, Etienne Laurin notifications@github.comwrote:

@rlshepherd https://github.com/rlshepherd We have just released
RethinkDB 1.12.2, which includes a fix for this issue.

Reply to this email directly or view it on GitHubhttps://github.com//issues/2222#issuecomment-40016403
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srh srh

Labels
pr:high tp:bug
Projects
None yet
Milestone
1.12.2
Development

No branches or pull requests
5 participants
@coffeemug @srh @danielmewes @AtnNn @rlshepherd