Unit tests hang

[ghost]

Unit tests hang during RDBBtree.SindexInterruptionViaDrop on CentOS 6.4 inside OpenVZ container.
The problem with reproducibility is, on some machines (like on the bare metal server with Xeon) it always hangs, but the same server (set up by scripts) inside VirtualBox did not hang (all tests passed).

[coffeemug]

@Tryneus -- could you take a look at this when you get the chance?

[Tryneus]

Sure, but it could take some time.

[AtnNn]

Moving to 1.7.x

[Tryneus]

Ok, managed to reproduce this in a CentOS 6.4 VM, doesn't happen every time, but I'll see what I can find.

[Tryneus]

Ok, seriously looking at this now, for real.

[Tryneus]

Ran into this crash when running the debug build unittests, may be related:

[ RUN      ] RDBBtree.SindexInterruptionViaDrop
Version: rethinkdb 1.6.1-fallback (debug) (GCC 4.4.7)
error: Error in ../src/arch/io/disk.cc at line 73:
error: Assertion failed: [outstanding_txn == 0] Closing a file with outstanding txns
error: Backtrace:
error: Tue Jul  9 17:50:24 2013

       1: rethinkdb_backtrace(void**, int) at thread_stack_pcs.cc:151
       2: lazy_backtrace_t::lazy_backtrace_t() at backtrace.cc:250
       3: format_backtrace(bool) at backtrace.cc:197
       4: report_fatal_error(char const*, int, char const*, ...) at errors.cc:67
       5: linux_disk_manager_t::~linux_disk_manager_t() at disk.cc:73
       6: linux_disk_manager_t::~linux_disk_manager_t() at disk.cc:74
       7: scoped_ptr_t<linux_disk_manager_t>::reset() at scoped.hpp:66
       8: scoped_ptr_t<linux_disk_manager_t>::~scoped_ptr_t() at scoped.hpp:30
       9: io_backender_t::~io_backender_t() at disk.cc:177
       10: unittest::run_sindex_interruption_via_drop_test() at rdb_btree.cc:419
       11: boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) at function_template.hpp:113
       12: boost::function0<void>::operator()() const at function_template.hpp:1014
       13: starter_t::run_wrapper(boost::function<void ()()> const&) at runtime.cc:53
       14: boost::_mfi::mf1<void, starter_t, boost::function<void ()()> const&>::operator()(starter_t*, boost::function<void ()()> const&) const at mem_fn_template.hpp:163
       15: void boost::_bi::list2<boost::_bi::value<starter_t*>, boost::_bi::value<boost::function<void ()()> > >::operator()<boost::_mfi::mf1<void, starter_t, boost::function<void ()()> const&>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, starter_t, boost::function<void ()()> const&>&, boost::_bi::list0&, int) at bind.hpp:307
       16: boost::_bi::bind_t<void, boost::_mfi::mf1<void, starter_t, boost::function<void ()()> const&>, boost::_bi::list2<boost::_bi::value<starter_t*>, boost::_bi::value<boost::function<void ()()> > > >::operator()() at bind_template.hpp:21
       17: boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, starter_t, boost::function<void ()()> const&>, boost::_bi::list2<boost::_bi::value<starter_t*>, boost::_bi::value<boost::function<void ()()> > > >, void>::invoke(boost::detail::function::function_buffer&) at function_template.hpp:154
       18: boost::function0<void>::operator()() const at function_template.hpp:1014
       19: callable_action_instance_t<boost::function<void ()()> >::run_action() at callable_action.hpp:28
       20: callable_action_wrapper_t::run() at runtime_utils.cc:67
       21: coro_t::run() at coroutines.cc:178

Also, I'm wondering if #1155 is related.

[coffeemug]

#1155 is confirmed to only happen on CentOS builds because of a different vector implementation in the standard libs, so it's unlikely to be the culprit.

[anatol]

RDBProtocol.OvershardedSindexCreateDrop has the same problem on my Linux Arch.

[coffeemug]

According to @Tryneus, this is likely a dup of #1084, so reassigning to @srh.

[danielmewes]

A probable fix for this has been merged into the next and v1.11.x branches (#1084 (comment) ).

@anatol @Herby if you get a change to re-test this with the current code, please let me know if it works or not.

[AtnNn]

Punting to 1.12.x

[coffeemug]

Moving to subsequent on the assumption that only critical bugs should be in the point release, and if something has been open for months, it isn't critical.

[danielmewes]

Closing, because it hasn't been reported anymore since the fix for this related issue #1084 (comment) .

Please re-open if the problem is observed again.

[anatol]

Sorry for the delay. I confirm that I do not see any test failures reported previously.

Thanks for fixing it.

[danielmewes]

@anatol Oh hey, thank you for confirming this. :-)