Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New failures when running with real Parsec (mock_serialise) #1734

Closed
jeanphilippeD opened this issue Aug 7, 2019 · 1 comment · Fixed by #1761

Comments

@jeanphilippeD
Copy link
Contributor

commented Aug 7, 2019

This was not an issue a few weeks back when we ran tests.

---- mock_network::multi_split stdout ----
thread 'mock_network::multi_split' panicked at 'update_with_genesis_related_info different our_infos:
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 84c37d..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ee162..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 899444..), PublicId(name: 92e316..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..), PublicId(name: ded392..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..)] }), (SectionInfo(prefix: Prefix(10), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..), PublicId(name: ded392..)] })],
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 84c37d..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ee162..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 40804d..), PublicId(name: 4ae402..), PublicId(name: 4ee162..), PublicId(name: 713414..), PublicId(name: 8395bb..), PublicId(name: 84c37d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 899444..), PublicId(name: 92e316..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: c39e0d..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..), PublicId(name: ded392..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..)] }), (SectionInfo(prefix: Prefix(10), members: {PublicId(name: 8395bb..), PublicId(name: 84c37d..), PublicId(name: 899444..), PublicId(name: 92e316..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 8395bb..), PublicId(name: 92e316..), PublicId(name: c39e0d..), PublicId(name: d3b0c0..), PublicId(name: ded392..)] })]', src/chain/shared_state.rs:77:17
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

=====================================================
Some([2854454551, 1570528927, 1481508013, 892138905])
=====================================================


---- mock_network::multiple_joining_nodes stdout ----
thread 'mock_network::multiple_joining_nodes' panicked at 'update_with_genesis_related_info different our_infos:
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 9c58d3..), PublicId(name: d50fc9..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 43357b..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..)] }), (SectionInfo(prefix: Prefix(00), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 26de3a..), PublicId(name: 31b1d7..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 43357b..), PublicId(name: 4e9afa..)] })],
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9c58d3..), PublicId(name: bd5419..), PublicId(name: d50fc9..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 9c58d3..), PublicId(name: d50fc9..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 4e9afa..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(0), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 43357b..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 4e9afa..), PublicId(name: 5c3542..)] }), (SectionInfo(prefix: Prefix(00), members: {PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 26de3a..), PublicId(name: 31b1d7..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 17a582..), PublicId(name: 31b1d7..), PublicId(name: 43357b..), PublicId(name: 4e9afa..)] })]', src/chain/shared_state.rs:77:17

---- mock_network::simultaneous_joining_nodes_three_section_with_one_ready_to_split stdout ----
thread 'mock_network::simultaneous_joining_nodes_three_section_with_one_ready_to_split' panicked at 'update_with_genesis_related_info different our_infos:
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: aaec25..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: b88bdd..)] })],
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 873e06..), PublicId(name: b88bdd..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: aaec25..), PublicId(name: b88bdd..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 8605b9..), PublicId(name: 873e06..), PublicId(name: 9e73ec..), PublicId(name: b88bdd..)] })]', src/chain/shared_state.rs:77:17

---- mock_network::more_than_section_size_nodes stdout ----
thread 'mock_network::more_than_section_size_nodes' panicked at 'update_with_genesis_related_info different our_infos:
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 18c1d8..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: ec49ae..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 883361..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: ec49ae..)] }), (SectionInfo(prefix: Prefix(10), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 13), ProofSet { [PublicId(name: 883361..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..)] })],
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 18c1d8..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 31b1d7..), PublicId(name: 4f308a..), PublicId(name: 6fe5cf..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: bd5419..), PublicId(name: dc8c5e..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: ec49ae..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..), PublicId(name: ec49ae..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..)] }), (SectionInfo(prefix: Prefix(10), members: {PublicId(name: 883361..), PublicId(name: 9a5fce..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..)}, prev_hash_len: 1, version: 13), ProofSet { [PublicId(name: 883361..), PublicId(name: a70fcd..), PublicId(name: b070a4..), PublicId(name: bd5419..), PublicId(name: d1bc87..), PublicId(name: dc8c5e..)] })]', src/chain/shared_state.rs:77:17

---- mock_network::check_section_info_ack stdout ----
thread 'mock_network::check_section_info_ack' panicked at 'update_with_genesis_related_info different our_infos:
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: c1db3d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: ed249d..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 13), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 14), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: ed249d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 15), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: f38659..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: aa1dbc..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 16), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: c42f42..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a458e7..), PublicId(name: a70a04..), PublicId(name: aa1dbc..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 17), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: aa1dbc..), PublicId(name: c1db3d..), PublicId(name: ed249d..), PublicId(name: f38659..)] })],
[(SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..)}, prev_hash_len: 0, version: 0), ProofSet { [] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..)}, prev_hash_len: 1, version: 1), ProofSet { [PublicId(name: 0575d2..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 2), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 3), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 40804d..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 4), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 5), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 6), ProofSet { [PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 7), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)}, prev_hash_len: 1, version: 8), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..)}, prev_hash_len: 1, version: 9), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)}, prev_hash_len: 1, version: 10), ProofSet { [PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 8436a6..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)}, prev_hash_len: 1, version: 11), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 713414..), PublicId(name: c1db3d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: ed249d..)}, prev_hash_len: 1, version: 12), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 13), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 35b773..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 14), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: ed249d..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 15), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: f38659..)] }), (SectionInfo(prefix: Prefix(), members: {PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: aa1dbc..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 16), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1059e7..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a70a04..), PublicId(name: c42f42..)] }), (SectionInfo(prefix: Prefix(1), members: {PublicId(name: 825264..), PublicId(name: 8436a6..), PublicId(name: a458e7..), PublicId(name: a70a04..), PublicId(name: aa1dbc..), PublicId(name: c1db3d..), PublicId(name: c42f42..), PublicId(name: ed249d..), PublicId(name: f38659..)}, prev_hash_len: 1, version: 17), ProofSet { [PublicId(name: 0575d2..), PublicId(name: 1d5c97..), PublicId(name: 202467..), PublicId(name: 35b773..), PublicId(name: 40804d..), PublicId(name: 4ff5a2..), PublicId(name: 70a45b..), PublicId(name: 713414..), PublicId(name: 825264..), PublicId(name: aa1dbc..), PublicId(name: ed249d..), PublicId(name: f38659..)] })]', src/chain/shared_state.rs:77:17

---- mock_network::cache::response_caching stdout ----
thread 'mock_network::cache::response_caching' panicked at 'poll_and_resend has been called 1000 times.', tests/mock_network/utils.rs:418:5


failures:
    mock_network::cache::response_caching
    mock_network::check_section_info_ack
    mock_network::more_than_section_size_nodes
    mock_network::multi_split
    mock_network::multiple_joining_nodes
    mock_network::simultaneous_joining_nodes_three_section_with_one_ready_to_split
@maqi

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

The reason of the failure is because the sequence of the proof that we got from parsec_poll is different among nodes.
For the same section_info, all nodes voted the same.
However, in one node(efde4e), the proofs come in with the sequence such as: 00,0c, cd, ef, cb, ec
Meanwhile, the sequence in other nodes is: 00,0c, cd, ec, ef, cb
Although all nodes' proofs arrived on all nodes, due to the fact that the chain::poll will be called before the last proof come in and consensus only requires quorum, this makes that node having a different proof set of that section_info version to other nodes'.
When later on that node receives our_info from other nodes, the panic will be fired because of the diff in the proof_set.

Here is the link of the debugging commit to re-generate the failure and presented the logging evidence (proofs_come_in_with_different_order.txt ). Which shows the node efde4e has proofs came in a different order to other nodes.

@jeanphilippeD jeanphilippeD moved this from To Do to In progress in Secure Message Delivery (SMD) Aug 19, 2019

@jeanphilippeD jeanphilippeD moved this from In progress to Needs review in Secure Message Delivery (SMD) Aug 28, 2019

@jeanphilippeD jeanphilippeD moved this from Needs review to Approved by reviewer in Secure Message Delivery (SMD) Aug 29, 2019

@jeanphilippeD jeanphilippeD moved this from Approved by reviewer to In progress in Secure Message Delivery (SMD) Aug 29, 2019

Secure Message Delivery (SMD) automation moved this from In progress to Merged / Done Aug 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.