[Bug] The blockchain stucks when there are some malicious nodes

[fCorleone]

🐛 Bug

I start a swarm with 10 nodes and I set 3 of the nodes as malicious ones. The malicious node can modify the values of some fields of sending packets to make a chaos in the system. However, I found that all the nodes stop producing new blocks after reaching to the height of 358. And the mint command reports some errors. I think some transactions cannot be processed correctly. I attached the log files too.

To reproduce

1. Start a new swarm with 10 nodes by using diem-swarm tool.
2. Start to mint by

cluster-test --mint-file "/home/fuchen/Desktop/SCALE_DIEM/mint.key" --swarm --peers "localhost:44289,localhost:39251,localhost:36947,localhost:38273,localhost:41187,localhost:35033,localhost:36867,localhost:37149,localhost:33861,localhost:37105" --emit-tx --workers-per-ac 1

3. The bug occurs

Stack trace/error message

2022-04-05T11:24:59.232522Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:698 [Client { url: "http://localhost:38273/v1", inner: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*"}, timeout: 10s }, state: StateManager { last_known_state: Mutex { data: Some(State { chain_id: 4, version: 4347, timestamp_usecs: 1649157859030073 }), poisoned: false, .. } }, retry: Retry { max_retries: 20, delay: 500ms } }] Failed to submit request: Error { inner: Inner { kind: Timeout, source: Some(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(38273), path: "/v1", query: None, fragment: None }, source: TimedOut }), json_rpc_error: None } }
2022-04-05T11:24:59.405464Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:698 [Client { url: "http://localhost:37105/v1", inner: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*"}, timeout: 10s }, state: StateManager { last_known_state: Mutex { data: Some(State { chain_id: 4, version: 4347, timestamp_usecs: 1649157859030073 }), poisoned: false, .. } }, retry: Retry { max_retries: 20, delay: 500ms } }] Failed to submit request: Error { inner: Inner { kind: Timeout, source: Some(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(37105), path: "/v1", query: None, fragment: None }, source: TimedOut }), json_rpc_error: None } }

And after a period, it shows:

2022-04-05T11:27:44.240016Z [tokio-runtime-worker] INFO testsuite/cluster-test/src/tx_emitter.rs:726 [Client { url: "http://localhost:44289/v1", inner: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*"}, timeout: 10s }, state: StateManager { last_known_state: Mutex { data: Some(State { chain_id: 4, version: 4347, timestamp_usecs: 1649157859030073 }), poisoned: false, .. } }, retry: Retry { max_retries: 20, delay: 500ms } }] Transactions were not committed before expiration: [(02C5EE6BBDD7251D19EE943911184704, 0), (6796EBA02896E4FE4801EFF096E397F1, 0), (8F1F70889CB96E6C4925C3B301046024, 0), (39423FEA1B5F0A43A1BC2E5EF1F75A38, 0), (BC400A390E191174DB7B8E8CF6B4477F, 0), (A64D6206D2DE57BCE17FFA178423BD7C, 0), (05263070B5B3B4F895099F4168BCD595, 0), (2E57B69C5BC8F991E017DA147CAA650D, 0), (8925B6B3D1E4B610355EA28D5281D98F, 0), (C5026F0E19BE7858792223625A345496, 0), (CCFDEFABF423302F9A0033915F034385, 0), (86B661ED2D00DB36CE66A58FFD3057B0, 0), (A6247026D5390833846A0D4C365733C3, 0), (D777B02EF55645CA86EAECA7839A7D39, 0), (49EB8802D9025C9672B8947CCC408521, 0)]
2022-04-05T11:27:44.715434Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 38F7C8431B6AB4F8EEFA8D2C1ED97544, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715489Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 8EEA6083920093C891F7FD36D5208251, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715499Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account AD2DCAEA497103AC58C81097687969E9, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715506Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 897EF36A23B68C3C0F7303FB6BDAAB0D, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715514Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account B662EBF7E65A294649AB5AC0FBDE7D77, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715522Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 4AC9AEDFE69BEB4A57662A794D2D1EFA, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715530Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 7AFAB348D11720F5C15E24C534A747E0, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715538Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account C064853F2CB399080D04C4462DD4BF3D, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715546Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 15C1515E0C6EBECCBA689963DBD34261, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715554Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account C05C1D5068E7D673A4B8AB5D1ABE3F54, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715562Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 4792927364A42E385AC0E71AFE88959F, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715569Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 0F3484843DB7455CCEA1354FBF617CD4, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715577Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 2A34117AA9D1D6203FF14A65E673BD33, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715584Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account A7A4F4AD3E4AF72E7EC84AE5D30B375B, expected sequence 1, got from server: 0
2022-04-05T11:27:44.715595Z [tokio-runtime-worker] WARN testsuite/cluster-test/src/tx_emitter.rs:811 Wait deadline exceeded for account 53D89E01F4AC22AA5BA58583916C0634, expected sequence 1, got from server: 0

Expected Behavior

I think all the transactions should be processed correctly under this situation.

System information

Please complete the following information:

• Diem: branch testnet
• Rust version: cargo 1.61.0-nightly
• Ubuntu 20.04

Additional context

Log Files

0.zip
1.zip
2.zip
3.zip
4.zip
5.zip
6.zip
7.zip
8.zip
9.zip