Collect all intermittently failing tests

[rkeene]

If you have a test that fails intermittently let's collect them here for systematic investigation

[PlasmaPower]

On master 37de74f

[ RUN      ] rpc.online_reps
/home/lee/programming/cpp/raiblocks2/rai/core_test/rpc.cpp:3662: Failure
Expected: (nullptr) != (receive), actual: 8-byte object <00-00 00-00 00-00 00-00> vs 16-byte object <00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
[  FAILED  ] rpc.online_reps (238 ms)

EDIT: This test now fails with:

[ RUN      ] rpc.online_reps
nano/rpc_test/rpc.cpp:6121: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

[rkeene]

[ RUN      ] node.fork_publish
/home/rkeene/devel/raiblocks/nano/core_test/node.cpp:643: Failure
Value of: node1.active.roots.size ()
  Actual: 0
Expected: 1
[  FAILED  ] node.fork_publish (668 ms)

and

[ RUN      ] node.fork_multi_flip
/home/rkeene/devel/raiblocks/nano/core_test/node.cpp:788: Failure
Value of: node2.active.roots.size ()
  Actual: 1
Expected: 2
[  FAILED  ] node.fork_multi_flip (1820 ms)

[wezrule]

[ RUN      ] node.broadcast_elected
Assertion (!error) failed c:\users\wesley\documents\raiblocks\nano\node\node.cpp:3114

Failing on this release_assert in nano::active_transactions::add:

auto error (nano::work_validate (*block_a, &difficulty));
release_assert (!error);

[wezrule]

[ RUN      ] rpc.work_peer_bad
d:\agent\_work\2\s\src\vctools\crt\crtw32\stdcpp\thr\mutex.c(51): mutex destroyed while busy

Stack trace

[wezrule]

[ RUN      ] node.vote_by_hash_republish
/workspace/nano/core_test/node.cpp:2196: Failure
Value of: system.poll ()
  Actual: Deadline expired

[wezrule]

Failed with MacOS actions (https://github.com/nanocurrency/nano-node/runs/617432809?check_suite_focus=true)

[ RUN      ] node.aggressive_flooding
/Users/runner/runners/2.169.1/work/nano-node/nano-node/nano/core_test/node.cpp:3727: Failure
Expected equality of these values:
  1 + 2 * nodes_wallets.size () + 2
    Which is: 13
  node1.ledger.cache.block_count
    Which is: 12
[  FAILED  ] node.aggressive_flooding (2362 ms)

Also failed locally on my Windows system

[wezrule]

Failed with gcc actions (https://github.com/nanocurrency/nano-node/runs/616900835?check_suite_focus=true)

timeout 360 ./qt_test
QXcbConnection: Could not connect to display :2
timeout: the monitored command dumped core
./ci/test.sh: line 23: 42719 Aborted
DISPLAY=:2 ${TIMEOUT_CMD} ${TIMEOUT_TIME_ARG} ${TIMEOUT_SEC-${TIMEOUT_DEFAULT}} $@

[guilhermelawless]

Failed with clang actions

2020-04-28T13:30:04.6017646Z [ RUN      ] rpc.confirmation_active
2020-04-28T13:30:04.7142579Z unknown file: Failure
2020-04-28T13:30:04.7142983Z C++ exception with description "Address already in use" thrown in the test body.
2020-04-28T13:30:04.7143121Z [  FAILED  ] rpc.confirmation_active (114 ms)

[wezrule]

active_transactions.vote_replays
/workspace/nano/core_test/active_transactions.cpp:774: Failure
Expected equality of these values:
  1
  node.active.size ()
    Which is: 0

[cryptocode]

After another few hours of core_test looping:

[ RUN      ] active_transactions.pessimistic_elections
nano/core_test/active_transactions.cpp:1535: Failure
Expected equality of these values:
  1
  election_count
    Which is: 2

NOTE: A bit of investigation shows this is because confirm_expired_frontiers_pessimistically is not locked internally, so the test corrupts expired_optimistic_election_infos. Wesley points out it's not an issue for the production node as it's only called from the request loop thread. Possible fixes include internal locking, a separate mutex or some other workaround for the test.

[cryptocode]

[ RUN      ] node.bidirectional_tcp
Assertion (wallets.reps ().voting > 0) failed
void nano::request_aggregator::add(std::shared_ptr<nano::transport::channel> &, const std::vector<std::pair<nano::block_hash, nano::root> > &)
nano/node/request_aggregator.cpp:35

 0# nano::generate_stacktrace() in build/RelWithDebInfo/core_test
 1# assert_internal(char const*, char const*, char const*, unsigned int, bool) in build/RelWithDebInfo/core_test
 2# nano::request_aggregator::add(std::__1::shared_ptr<nano::transport::channel>&, std::__1::vector<std::__1::pair<nano::block_hash, nano::root>, std::__1::allocator<std::__1::pair<nano::block_hash, nano::root> > > const&) in build/RelWithDebInfo/core_test
 3# (anonymous namespace)::network_message_visitor::confirm_req(nano::confirm_req const&) in build/RelWithDebInfo/core_test
 4# nano::confirm_req::visit(nano::message_visitor&) const in build/RelWithDebInfo/core_test
 5# nano::network::process_message(nano::message const&, std::__1::shared_ptr<nano::transport::channel>) in build/RelWithDebInfo/core_test
 6# nano::transport::tcp_channels::process_message(nano::message const&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&, nano::public_key const&, std::__1::shared_ptr<nano::socket>, nano::bootstrap_server_type) in build/RelWithDebInfo/core_test
 7# nano::transport::tcp_channels::process_messages() in build/RelWithDebInfo/core_test
 8# nano::network::network(nano::node&, unsigned short)::$_2::operator()() const in build/RelWithDebInfo/core_test
 9# boost::detail::thread_data<nano::network::network(nano::node&, unsigned short)::$_2>::run() in build/RelWithDebInfo/core_test

[cryptocode]

[ RUN      ] node.dependency_graph_frontier
nano/core_test/node.cpp:4282: Failure
Value of: system.poll ()
  Actual: Deadline expired
Expected:

[LenWilliamson]

I just run the test in the top level nano_node directory this way:
cmake -D NANO_TEST=ON
make
and then I got all the test files in nano_node. When running ./core_test I get

[wezrule]

Thanks, what OS is this? If you still have the output, would be nice to see where it's failing

[LenWilliamson]

Operating System: Linux Mint 20.1
Kernel: Linux 5.4.0-65-generic
Architecture: x86-64

After running the test again I didn't get the output. But I don't know (yet) how to check where it's failing. What should I do to check it?

[wezrule]

no worries, if it happens again you can scroll up (assuming there's enough lines configured in terminal) to where the test is and it should output something when it fails

[LenWilliamson]

I see, so this is the output I got

[ RUN      ] network.last_contacted
/home/len/Crypto/nano-node/nano/core_test/network.cpp:153: Failure
Expected: (timestamp_after_keepalive) > (timestamp_before_keepalive), actual: 8-byte object <71-AD 89-35 87-02 00-00> vs 8-byte object <71-AD 89-35 87-02 00-00>
[  FAILED  ] network.last_contacted (212 ms)
[ RUN      ] network.multi_keepalive

Btw, what is the goal of this test and why has the timestep_before_keepalive be strictly greater? What is the "keepalive"?

[cryptocode]

Happens on CI now in rpc_test:

[ RUN      ] rpc.confirmation_active
Assertion (block->has_sideband ()) failed
void nano::blocks_confirm(nano::node &, const std::vector<std::shared_ptr<nano::block> > &)
/workspace/nano/node/testing.cpp:212

 0# nano::generate_stacktrace[abi:cxx11]() at /workspace/nano/lib/utility.cpp:79
 1# assert_internal(char const*, char const*, char const*, unsigned int, bool, std::basic_string_view<char, std::char_traits<char> >) at /workspace/nano/lib/utility.cpp:125
 2# nano::blocks_confirm(nano::node&, std::vector<std::shared_ptr<nano::block>, std::allocator<std::shared_ptr<nano::block> > > const&) at /workspace/nano/node/testing.cpp:213
 3# rpc_confirmation_active_Test::TestBody() at /workspace/nano/rpc_test/rpc.cpp:8050
 4# void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2443
 5# void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2479
 6# testing::Test::Run() in ./rpc_test
 7# testing::TestInfo::Run() in ./rpc_test
 8# testing::TestCase::Run() at /workspace/gtest/googletest/src/gtest.cc:2810
 9# testing::internal::UnitTestImpl::RunAllTests() at /workspace/gtest/googletest/src/gtest.cc:5176
10# bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2443
11# bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) at /workspace/gtest/googletest/src/gtest.cc:2479
12# testing::UnitTest::Run() at /workspace/gtest/googletest/src/gtest.cc:4786
13# RUN_ALL_TESTS() at /workspace/gtest/googletest/include/gtest/gtest.h:2341
14# main at /workspace/nano/rpc_test/entry.cpp:17
15# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
16# _start in ./rpc_test

[LenWilliamson]

When I run the ./core_test I got

After running ./core_test again, this test passed.