Mozart2 hangs

[layus]

Mozart2 sometimes hangs instead of exiting when all oz code is finished.
This occurs with gui applications (because of the ozwish pipe not closing properly) and also randomly on other programs.
It was already reported for "vm" tests.

[eregon]

That's odd, I remember the vm tests were passing reliably when I introduced them.
Maybe it's related to the test harness?
I guess ozwish is not involved when running those tests, right?

I'd try just commenting out the hanging test if you can reproduce, maybe it's just one of them having buggy termination.

[layus]

Ozwish is indeed not involved. And it is most probably one vm that does not close properly. If the ozemulator can hang randomly, then I guess that N vms will hang N-times more randomly-often ;-).

[layus]

@eregon The issue is quite easy to reproduce with parallel ozemulator vm.ozf ::: $(seq 1 100).

With gdb, you get

(gdb) bt
#0  0x00007f7d4f4fe567 in epoll_wait () from /nix/store/83lrbvbmxrgv7iz49mgd42yvhi473xp6-glibc-2.27/lib/libc.so.6
#1  0x0000000000526991 in boost::asio::detail::epoll_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) ()
#2  0x000000000052653a in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) ()
#3  0x0000000000526171 in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x000000000050c179 in main ()
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f7d514a1780 (LWP 17306) "ozemulator" 0x00007f7d4f4fe567 in epoll_wait () from /nix/store/83lrbvbmxrgv7iz49mgd42yvhi473xp6-glibc-2.27/lib/libc.so.6
(gdb)

Does it mean something to you ?

[eregon]

So that sounds like it's waiting on an open IO, maybe simply stdin?
I forgot how Mozart2 deals with that, it's been a while (but I thought waiting on stdin is the case in Mozart 1, not in Mozart 2).
I do remember there is only one IO thread which is the main/initial thread.

[layus]

To me it looks more like a bug/misuse of boost asio. There remains only one blocked thread, with nothing to wake it up. It looks like a worker thread waiting on an empty work queue.

There are several simmilar issues like https://stackoverflow.com/questions/41804866/asio-on-linux-stalls-in-epoll https://svn.boost.org/trac10/ticket/11069

[layus]

Funny enough, running vm.ozf like I did above does nothing except declaring a big record. In fact I am able to reproduce the issue with

functor
define
   skip
end

if I run enough instance in parallel. Apparently there needs to be some heavy load on the machine. Looks more and more like a race condition around asio usage.

[eregon]

@layus Which Boost version are you using?
For instance chriskohlhoff/asio#180 seems to say Boost 1.65.0 would have the fix for the first bug report.

[layus]

@eregon boost 1.67.

clang -fthread-sanitizer to the rescue! Does it mean something to you ?

==================
WARNING: ThreadSanitizer: data race (pid=3782)
  Read of size 1 at 0x7b8400001228 by thread T1:
    #0 boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::system::error_code&) /build/source/build/vm/boostenv/main/include/boost/asio/detail/deadline_timer_service.hpp:135:15 (ozemulator+0x633429)
    #1 boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::cancel() /build/source/build/vm/boostenv/main/include/boost/asio/basic_deadline_timer.hpp:310 (ozemulator+0x633429)
    #2 mozart::boostenv::BoostVM::run() /build/source/vm/boostenv/main/boostvm.cc:122 (ozemulator+0x633429)
    #3 main::$_0::operator()(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool) const /build/source/boosthost/emulator/emulator.cc:370:15 (ozemulator+0x5c4a19)
    #4 std::_Function_handler<bool (mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool), main::$_0>::_M_invoke(std::_Any_data const&, mozart::VirtualMachine*&&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/std_function.h:301 (ozemulator+0x5c4a19)
    #5 std::function<bool (mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)>::operator()(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool) const /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/std_function.h:706:14 (ozemulator+0x632ac1)
    #6 mozart::boostenv::BoostVM::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool) /build/source/vm/boostenv/main/boostvm.cc:86 (ozemulator+0x632ac1)
    #7 boost::_mfi::mf2<void, mozart::boostenv::BoostVM, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool>::operator()(mozart::boostenv::BoostVM*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool) const /build/source/build/vm/boostenv/main/include/boost/bind/mem_fn_template.hpp:280:29 (ozemulator+0x642555)
    #8 void boost::_bi::list3<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, boost::_bi::value<bool> >::operator()<boost::_mfi::mf2<void, mozart::boostenv::BoostVM, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, mozart::boostenv::BoostVM, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool>&, boost::_bi::list0&, int) /build/source/build/vm/boostenv/main/include/boost/bind/bind.hpp:398 (ozemulator+0x642555)
    #9 boost::_bi::bind_t<void, boost::_mfi::mf2<void, mozart::boostenv::BoostVM, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool>, boost::_bi::list3<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, boost::_bi::value<bool> > >::operator()() /build/source/build/vm/boostenv/main/include/boost/bind/bind.hpp:1294 (ozemulator+0x642555)
    #10 boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf2<void, mozart::boostenv::BoostVM, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool>, boost::_bi::list3<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>, boost::_bi::value<bool> > > >::run() /build/source/build/vm/boostenv/main/include/boost/thread/detail/thread.hpp:117 (ozemulator+0x642555)
    #11 thread_proxy <null> (libboost_thread.so.1.67.0+0x14f7c)

  Previous write of size 1 at 0x7b8400001228 by main thread:
    #0 boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::system::error_code&) /build/source/build/vm/boostenv/main/include/boost/asio/detail/deadline_timer_service.hpp:145:35 (ozemulator+0x633c34)
    #1 boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::expires_at(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::posix_time::ptime const&, boost::system::error_code&) /build/source/build/vm/boostenv/main/include/boost/asio/detail/deadline_timer_service.hpp:193 (ozemulator+0x633c34)
    #2 boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::expires_at(boost::posix_time::ptime const&) /build/source/build/vm/boostenv/main/include/boost/asio/basic_deadline_timer.hpp:439 (ozemulator+0x633c34)
    #3 mozart::boostenv::BoostVM::onPreemptionTimerExpire(boost::system::error_code const&) /build/source/vm/boostenv/main/boostvm.cc:180 (ozemulator+0x633c34)
    #4 boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>::operator()(mozart::boostenv::BoostVM*, boost::system::error_code const&) const /build/source/build/vm/boostenv/main/include/boost/bind/mem_fn_template.hpp:165:29 (ozemulator+0x6432e6)
    #5 void boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()>::operator()<boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::rrlist1<boost::system::error_code const&> >(boost::_bi::type<void>, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>&, boost::_bi::rrlist1<boost::system::error_code const&>&, int) /build/source/build/vm/boostenv/main/include/boost/bind/bind.hpp:319 (ozemulator+0x6432e6)
    #6 void boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >::operator()<boost::system::error_code const&>(boost::system::error_code const&) /build/source/build/vm/boostenv/main/include/boost/bind/bind.hpp:1306 (ozemulator+0x6432e6)
    #7 boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code>::operator()() /build/source/build/vm/boostenv/main/include/boost/asio/detail/bind_handler.hpp:65 (ozemulator+0x6432e6)
    #8 void boost::asio::asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code> >(boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code>&, ...) /build/source/build/vm/boostenv/main/include/boost/asio/handler_invoke_hook.hpp:69 (ozemulator+0x6432e6)
    #9 void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> > >(boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code>&, boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >&) /build/source/build/vm/boostenv/main/include/boost/asio/detail/handler_invoke_helpers.hpp:37:3 (ozemulator+0x643166)
    #10 void boost::asio::detail::handler_work<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::asio::system_executor>::complete<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code> >(boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >, boost::system::error_code>&, boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> >&) /build/source/build/vm/boostenv/main/include/boost/asio/detail/handler_work.hpp:82 (ozemulator+0x643166)
    #11 boost::asio::detail::wait_handler<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mozart::boostenv::BoostVM, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<mozart::boostenv::BoostVM*>, boost::arg<1> (*)()> > >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /build/source/build/vm/boostenv/main/include/boost/asio/detail/wait_handler.hpp:72 (ozemulator+0x643166)
    #12 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /build/source/build/boosthost/emulator/include/boost/asio/detail/scheduler_operation.hpp:40:5 (ozemulator+0x5ec370)
    #13 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /build/source/build/boosthost/emulator/include/boost/asio/detail/impl/scheduler.ipp:401 (ozemulator+0x5ec370)
    #14 boost::asio::detail::scheduler::run(boost::system::error_code&) /build/source/build/boosthost/emulator/include/boost/asio/detail/impl/scheduler.ipp:154:10 (ozemulator+0x5eb936)
    #15 boost::asio::io_context::run() /build/source/build/boosthost/emulator/include/boost/asio/impl/io_context.ipp:62:24 (ozemulator+0x5c2245)
    #16 mozart::boostenv::BoostEnvironment::runIO() /build/source/boosthost/emulator/../../vm/boostenv/main/boostenv.hh:199 (ozemulator+0x5c2245)
    #17 main /build/source/boosthost/emulator/emulator.cc:423 (ozemulator+0x5c2245)

  Location is heap block of size 4944 at 0x7b8400000000 allocated by main thread:
    #0 operator new(unsigned long) <null> (ozemulator+0x5bcd76)
    #1 __gnu_cxx::new_allocator<std::_Fwd_list_node<mozart::boostenv::BoostVM> >::allocate(unsigned long, void const*) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/ext/new_allocator.h:111:27 (ozemulator+0x5eb720)
    #2 std::allocator_traits<std::allocator<std::_Fwd_list_node<mozart::boostenv::BoostVM> > >::allocate(std::allocator<std::_Fwd_list_node<mozart::boostenv::BoostVM> >&, unsigned long) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/alloc_traits.h:436 (ozemulator+0x5eb720)
    #3 std::_Fwd_list_base<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::_M_get_node() /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.h:337 (ozemulator+0x5eb720)
    #4 std::_Fwd_list_node<mozart::boostenv::BoostVM>* std::_Fwd_list_base<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::_M_create_node<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.h:345 (ozemulator+0x5eb720)
    #5 std::_Fwd_list_node_base* std::_Fwd_list_base<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::_M_insert_after<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(std::_Fwd_list_const_iterator<mozart::boostenv::BoostVM>, mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.tcc:59 (ozemulator+0x5eb720)
    #6 void std::forward_list<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::emplace_front<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.h:807 (ozemulator+0x5eb720)
    #7 mozart::boostenv::BoostEnvironment::addVM(long, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool, mozart::VirtualMachineOptions) /build/source/boosthost/emulator/../../vm/boostenv/main/boostenv.hh:128 (ozemulator+0x5eb720)
    #8 mozart::boostenv::BoostEnvironment::addInitialVM(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozart::VirtualMachineOptions) /build/source/boosthost/emulator/../../vm/boostenv/main/boostenv-decl.hh:78:12 (ozemulator+0x5c7c88)
    #9 main /build/source/boosthost/emulator/emulator.cc:422:12 (ozemulator+0x5c2218)

  Thread T1 (tid=3784, running) created by main thread at:
    #0 pthread_create <null> (ozemulator+0x531876)
    #1 boost::thread::start_thread_noexcept() <null> (libboost_thread.so.1.67.0+0x14089)
    #2 void __gnu_cxx::new_allocator<mozart::boostenv::BoostVM>::construct<mozart::boostenv::BoostVM, mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(mozart::boostenv::BoostVM*, mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/ext/new_allocator.h:136:23 (ozemulator+0x5eb75c)
    #3 void std::allocator_traits<std::allocator<mozart::boostenv::BoostVM> >::construct<mozart::boostenv::BoostVM, mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(std::allocator<mozart::boostenv::BoostVM>&, mozart::boostenv::BoostVM*, mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/alloc_traits.h:475 (ozemulator+0x5eb75c)
    #4 std::_Fwd_list_node<mozart::boostenv::BoostVM>* std::_Fwd_list_base<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::_M_create_node<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.h:351 (ozemulator+0x5eb75c)
    #5 std::_Fwd_list_node_base* std::_Fwd_list_base<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::_M_insert_after<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(std::_Fwd_list_const_iterator<mozart::boostenv::BoostVM>, mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.tcc:59 (ozemulator+0x5eb75c)
    #6 void std::forward_list<mozart::boostenv::BoostVM, std::allocator<mozart::boostenv::BoostVM> >::emplace_front<mozart::boostenv::BoostEnvironment&, long&, int, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool&>(mozart::boostenv::BoostEnvironment&, long&, int&&, mozart::VirtualMachineOptions&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&) /nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0/include/c++/7.3.0/bits/forward_list.h:807 (ozemulator+0x5eb75c)
    #7 mozart::boostenv::BoostEnvironment::addVM(long, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool, mozart::VirtualMachineOptions) /build/source/boosthost/emulator/../../vm/boostenv/main/boostenv.hh:128 (ozemulator+0x5eb75c)
    #8 mozart::boostenv::BoostEnvironment::addInitialVM(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozart::VirtualMachineOptions) /build/source/boosthost/emulator/../../vm/boostenv/main/boostenv-decl.hh:78:12 (ozemulator+0x5c7c88)
    #9 main /build/source/boosthost/emulator/emulator.cc:422:12 (ozemulator+0x5c2218)

SUMMARY: ThreadSanitizer: data race /build/source/build/vm/boostenv/main/include/boost/asio/detail/deadline_timer_service.hpp:135:15 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::system::error_code&)

[layus]

(just a note for my future self) This may be of interest: https://stackoverflow.com/questions/43168199/cancelling-boost-asio-deadline-timer-safely#43169596

[layus]

Bottom line: timers are not thread-safe. Because we restart the timer within the handler (which runs in the io thread) we must only access this handler from the io thread. See #307 for details.

Moreover, there is the possibility of a small race condition when cancelling a timer. It may appear that the cancel() does nothing if it occurs right after the timer expiration, but before it's restart in the onPreemptionTimerXXX handler.

I had to use the trick described in http://stackoverflow.com/questions/43168199/cancelling-boost-asio-deadline-timer-safely#43169596 to achieve a safe cancellation.

deleting the timer was also a puzzle, but I think I got it right.