Testcase "multiple_service_test" stalls

[hugoferreira]

After a clean install (virtualized Ubuntu Server 12.04.02), when running make test, the job stalls at [TESTCASE] multiple_service_test. There's no outstanding active process running in the machine, with the occasional bump in CPU from zookeeper.

[hugoferreira]

Sometimes it fails with the following error:

      [TESTCASE] multiple_service_test
                 multiple_service_test FAILED
Running 2 test cases...
Testing early connection...
got event SESSION state CONNECTED on path 
serviceClass = echo
getting serviceClass/echo/echo
Called connect to echo/echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0xbf2a10 data 0xbf23b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
type = DELETED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4b0000940 data 0xbf23b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
creating socket monitor at 0x7fff0d8f8320
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4b0000f70 data 0xbf23b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4b0000940 data 0xbf23b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
getting serviceClass/echo/echo
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo/tcp
Called connect to echo/echo
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo/tcp
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0xbf3da0 data 0xbf2420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Called connect to echo/echo
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2ab4b0000e60 data 0xbf2420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Called connect to echo/echo
echo service is listening on tcp://*:15000
got key echo/echo/tcp
connected to tcp://127.0.0.1:15000
connected to tcp://127.0.0.1:15000
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2ab4b0000940 data 0xbf2420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Checking that we are connected 
done.
shutting down socket monitor at 0x7fff0d8f8320
0 monitors still active
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2ab4b00009a0 data 0xbf2420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Assertion failed: errno == EAGAIN (socket_base.cpp:863)

----------------- Exception thrown ------------------------
type:   boost::detail::system_signal_exception
pid:    20928; tid: 20931
stack:
00: 0x0x2ab4a2d8fe07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x2ab4a2d8fe07]
01: 0x0x2ab4a34b163a at boost::execution_monitor::catch_signals(boost::unit_test::callback0<int> const&) + 0x11a in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4963a/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor13catch_signalsERKNS_9unit_test9callback0IiEE+0x11a) [0x2ab4a34b163a]
02: 0x0x2ab4a34b1efb at boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) + 0x2b in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49efb/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor7executeERKNS_9unit_test9callback0IiEE+0x2b) [0x2ab4a34b1efb]
03: 0x0x2ab4a34cd22b at boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) + 0xab in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6522b/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test19unit_test_monitor_t21execute_and_translateERKNS0_9test_caseE+0xab) [0x2ab4a34cd22b]
04: 0x0x2ab4a34b89a1 at boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) + 0xb1 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x509a1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14framework_impl5visitERKNS0_9test_caseE+0xb1) [0x2ab4a34b89a1]
05: 0x0x2ab4a34e9093 at boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) + 0x53 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x81093/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test18traverse_test_treeERKNS0_10test_suiteERNS0_17test_tree_visitorE+0x53) [0x2ab4a34e9093]
06: 0x0x2ab4a34b40d1 at boost::unit_test::framework::run(unsigned long, bool) + 0x551 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4c0d1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test9framework3runEmb+0x551) [0x2ab4a34b40d1]
07: 0x0x2ab4a34cc66f at boost::unit_test::unit_test_main(bool (*)(), int, char**) + 0xcf in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6466f/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc+0xcf) [0x2ab4a34cc66f]
08: 0x0x2ab4a3f3e76d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2176d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2ab4a3f3e76d]
09: 0x0x40ee21 in build/x86_64/tests/multiple_service_test + 0xee21build/x86_64/tests/multiple_service_test() [0x40ee21]


----------------- Exception thrown ------------------------
type:   boost::execution_exception
pid:    20928; tid: 20931
stack:
00: 0x0x2ab4a2d8fe07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x2ab4a2d8fe07]
01: 0x0x2ab4a34b18ec in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x498ec/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(+0x498ec) [0x2ab4a34b18ec]
02: 0x0x2ab4a34b199c in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4999c/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(+0x4999c) [0x2ab4a34b199c]
03: 0x0x2ab4a34b1b70 at boost::detail::system_signal_exception::report() const + 0x1d0 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49b70/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZNK5boost6detail23system_signal_exception6reportEv+0x1d0) [0x2ab4a34b1b70]
04: 0x0x2ab4a34b1f84 at boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) + 0xb4 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49f84/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor7executeERKNS_9unit_test9callback0IiEE+0xb4) [0x2ab4a34b1f84]
05: 0x0x2ab4a34cd22b at boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) + 0xab in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6522b/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test19unit_test_monitor_t21execute_and_translateERKNS0_9test_caseE+0xab) [0x2ab4a34cd22b]
06: 0x0x2ab4a34b89a1 at boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) + 0xb1 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x509a1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14framework_impl5visitERKNS0_9test_caseE+0xb1) [0x2ab4a34b89a1]
07: 0x0x2ab4a34e9093 at boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) + 0x53 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x81093/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test18traverse_test_treeERKNS0_10test_suiteERNS0_17test_tree_visitorE+0x53) [0x2ab4a34e9093]
08: 0x0x2ab4a34b40d1 at boost::unit_test::framework::run(unsigned long, bool) + 0x551 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4c0d1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test9framework3runEmb+0x551) [0x2ab4a34b40d1]
09: 0x0x2ab4a34cc66f at boost::unit_test::unit_test_main(bool (*)(), int, char**) + 0xcf in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6466f/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc+0xcf) [0x2ab4a34cc66f]
10: 0x0x2ab4a3f3e76d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2176d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2ab4a3f3e76d]
11: 0x0x40ee21 in build/x86_64/tests/multiple_service_test + 0xee21build/x86_64/tests/multiple_service_test() [0x40ee21]

unknown location(0): fatal error in "test_early_connection": signal: SIGABRT (application abort requested)
./soa/service/testing/multiple_service_test.cc(130): last checkpoint
got event SESSION state CONNECTED on path 
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4b0000f70 data 0xbf23b0
Starting multiple services test 
onServiceProvidersChanged(serviceClass/echo)
creating socket monitor at 0x2ab4c40044f0
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo0
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo0
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4c4000aa0 data 0x2ab4c4000ac0
onServiceProvidersChanged(serviceClass/echo)
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2ab4c8000940 data 0x2ab4c4000ac0
onServiceProvidersChanged(serviceClass/echo)
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo/tcp
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo/tcp
Called connect to echo0/echo
connected to echo0
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2ab4c8000cf0 data 0x2ab4c8000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo
Called connect to echo0/echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2ab4c8000940 data 0x2ab4c8000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo
Called connect to echo0/echo
echo service is listening on tcp://*:15000
got key echo0/echo/tcp
connected to tcp://127.0.0.1:15000
serviceClass/echo/echo0 --> {"serviceName":"echo0","servicePath":"echo0"}
echo0/echo/tcp --> [{"transports":[{"addr":"127.0.0.1","hostScope":"ubuntu-vm","name":"tcp","port":15000},{"name":"zeromq","socketType":6,"uri":"tcp://127.0.0.1:15000"}],"zmqConnectUri":"tcp://127.0.0.1:15000"},{"transports":[{"addr":"10.211.55.30","hostScope":"*","name":"tcp","port":15000},{"name":"zeromq","socketType":6,"uri":"tcp://10.211.55.30:15000"}],"zmqConnectUri":"tcp://10.211.55.30:15000"}]
shutting down
shutting down socket monitor at 0x2ab4c40044f0
0 monitors still active
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2ab4c8000cf0 data 0x2ab4c8000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo

*** 1 failure detected in test suite "Master Test Suite"
                       multiple_service_test FAILED
make: *** [build/x86_64/tests/multiple_service_test.passed] Error 1
bytter@ubuntu-vm:~/rtbkit-preview$ make test
      [TESTCASE] multiple_service_test
                 multiple_service_test FAILED
Running 2 test cases...
Testing early connection...
got event SESSION state CONNECTED on path 
serviceClass = echo
type = DELETED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x21c7a10 data 0x21c73b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
creating socket monitor at 0x7fff320a4a80
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2b5574000940 data 0x21c73b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2b5574000db0 data 0x21c73b0
******* CHANGE TO SERVICE NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo
serviceClass = echo
getting serviceClass/echo/echo
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo/tcp
Called connect to echo/echo
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo/tcp
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2b5574000d10 data 0x21c7420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Called connect to echo/echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2b5574000db0 data 0x21c7420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Called connect to echo/echo
echo service is listening on tcp://*:15000
got key echo/echo/tcp
connected to tcp://127.0.0.1:15000
connected to tcp://127.0.0.1:15000
Checking that we are connected 
done.
shutting down socket monitor at 0x7fff320a4a80
0 monitors still active
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo context 0x2b5574000d10 data 0x21c7420
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo/echo
Assertion failed: errno == EAGAIN (socket_base.cpp:863)

----------------- Exception thrown ------------------------
type:   boost::detail::system_signal_exception
pid:    20966; tid: 20969
stack:
00: 0x0x2b5565b31e07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x2b5565b31e07]
01: 0x0x2b556625363a at boost::execution_monitor::catch_signals(boost::unit_test::callback0<int> const&) + 0x11a in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4963a/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor13catch_signalsERKNS_9unit_test9callback0IiEE+0x11a) [0x2b556625363a]
02: 0x0x2b5566253efb at boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) + 0x2b in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49efb/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor7executeERKNS_9unit_test9callback0IiEE+0x2b) [0x2b5566253efb]
03: 0x0x2b556626f22b at boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) + 0xab in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6522b/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test19unit_test_monitor_t21execute_and_translateERKNS0_9test_caseE+0xab) [0x2b556626f22b]
04: 0x0x2b556625a9a1 at boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) + 0xb1 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x509a1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14framework_impl5visitERKNS0_9test_caseE+0xb1) [0x2b556625a9a1]
05: 0x0x2b556628b093 at boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) + 0x53 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x81093/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test18traverse_test_treeERKNS0_10test_suiteERNS0_17test_tree_visitorE+0x53) [0x2b556628b093]
06: 0x0x2b55662560d1 at boost::unit_test::framework::run(unsigned long, bool) + 0x551 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4c0d1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test9framework3runEmb+0x551) [0x2b55662560d1]
07: 0x0x2b556626e66f at boost::unit_test::unit_test_main(bool (*)(), int, char**) + 0xcf in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6466f/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc+0xcf) [0x2b556626e66f]
08: 0x0x2b5566ce076d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2176d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2b5566ce076d]
09: 0x0x40ee21 in build/x86_64/tests/multiple_service_test + 0xee21build/x86_64/tests/multiple_service_test() [0x40ee21]


----------------- Exception thrown ------------------------
type:   boost::execution_exception
pid:    20966; tid: 20969
stack:
00: 0x0x2b5565b31e07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x2b5565b31e07]
01: 0x0x2b55662538ec in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x498ec/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(+0x498ec) [0x2b55662538ec]
02: 0x0x2b556625399c in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4999c/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(+0x4999c) [0x2b556625399c]
03: 0x0x2b5566253b70 at boost::detail::system_signal_exception::report() const + 0x1d0 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49b70/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZNK5boost6detail23system_signal_exception6reportEv+0x1d0) [0x2b5566253b70]
04: 0x0x2b5566253f84 at boost::execution_monitor::execute(boost::unit_test::callback0<int> const&) + 0xb4 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x49f84/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost17execution_monitor7executeERKNS_9unit_test9callback0IiEE+0xb4) [0x2b5566253f84]
05: 0x0x2b556626f22b at boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) + 0xab in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6522b/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test19unit_test_monitor_t21execute_and_translateERKNS0_9test_caseE+0xab) [0x2b556626f22b]
06: 0x0x2b556625a9a1 at boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) + 0xb1 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x509a1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14framework_impl5visitERKNS0_9test_caseE+0xb1) [0x2b556625a9a1]
07: 0x0x2b556628b093 at boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) + 0x53 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x81093/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test18traverse_test_treeERKNS0_10test_suiteERNS0_17test_tree_visitorE+0x53) [0x2b556628b093]
08: 0x0x2b55662560d1 at boost::unit_test::framework::run(unsigned long, bool) + 0x551 in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x4c0d1/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test9framework3runEmb+0x551) [0x2b55662560d1]
09: 0x0x2b556626e66f at boost::unit_test::unit_test_main(bool (*)(), int, char**) + 0xcf in /home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0 + 0x6466f/home/bytter/local/lib/libboost_unit_test_framework.so.1.47.0(_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc+0xcf) [0x2b556626e66f]
10: 0x0x2b5566ce076d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2176d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2b5566ce076d]
11: 0x0x40ee21 in build/x86_64/tests/multiple_service_test + 0xee21build/x86_64/tests/multiple_service_test() [0x40ee21]

unknown location(0): fatal error in "test_early_connection": signal: SIGABRT (application abort requested)
./soa/service/testing/multiple_service_test.cc(130): last checkpoint
got event SESSION state CONNECTED on path 
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2b5574000940 data 0x21c73b0
Starting multiple services test 
onServiceProvidersChanged(serviceClass/echo)
creating socket monitor at 0x2b55880044f0
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo0
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo/echo0
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2b5588000aa0 data 0x2b5588000ac0
onServiceProvidersChanged(serviceClass/echo)
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/serviceClass/echo context 0x2b558c000940 data 0x2b5588000ac0
onServiceProvidersChanged(serviceClass/echo)
createNode for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo/tcp
createPath for /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo/tcp
Called connect to echo0/echo
connected to echo0
type = CREATED state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2b558c000cf0 data 0x2b558c000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo
Called connect to echo0/echo
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2b558c000940 data 0x2b558c000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo
Called connect to echo0/echo
echo service is listening on tcp://*:15000
got key echo0/echo/tcp
connected to tcp://127.0.0.1:15000
serviceClass/echo/echo0 --> {"serviceName":"echo0","servicePath":"echo0"}
echo0/echo/tcp --> [{"transports":[{"addr":"127.0.0.1","hostScope":"ubuntu-vm","name":"tcp","port":15000},{"name":"zeromq","socketType":6,"uri":"tcp://127.0.0.1:15000"}],"zmqConnectUri":"tcp://127.0.0.1:15000"},{"transports":[{"addr":"10.211.55.30","hostScope":"*","name":"tcp","port":15000},{"name":"zeromq","socketType":6,"uri":"tcp://10.211.55.30:15000"}],"zmqConnectUri":"tcp://10.211.55.30:15000"}]
shutting down
shutting down socket monitor at 0x2b55880044f0
0 monitors still active
type = CHILD state = ZOO_CONNECTED_STATE path = /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo context 0x2b558c000cf0 data 0x2b558c000b50
******* CHANGE TO ENDPOINT NODE /dev/ubuntu-vm/home/bytter/rtbkit-preview_multiple_service_test/echo0/echo

*** 1 failure detected in test suite "Master Test Suite"
                       multiple_service_test FAILED
make: *** [build/x86_64/tests/multiple_service_test.passed] Error 1

[sunilrottoo]

Hi Hugo,

Thanks for reporting this. This is an issue we are currently investigating related to a race condition at shutdown with zeromq and ZooKeeper. The support for ZooKeeper is fairly new and we are seeing similar issues intermittently in production. We will let you know once this is fixed. Comments about the design, architecture and code are also most welcome!

Best,

Sunil

[EricRobert]

Hi Hugo,

I think I isolated the issue. I pushed a fix that should make things better but it's only a workaround. I've post my issue on the zeromq mailing list and I'll keep digging.

Note that this issue only happens on shutdown and thus, we've never bump into it in production.

Thanks again for reporting this issue and let me know :)

Éric
PS: I'll keep this issue open for now.

[hugoferreira]

No avail. I did git pull, make compile, make test, but it still stalls on the same test. Do you reckon I need to recompile the whole stack?

[EricRobert]

Did you do 'git submodule update' after you did 'git pull' ?

[hugoferreira]

😅 All ok, folks 👍