I think I know why testnet-1 has halted!

[alkadeta]

So this morning at least 20 or 30 validators whom have been using Google Cloud Provider have been banned and their virtual machines which their kira nodes have been running on, got shutdown by Google support. Reasoning was Cryptocurrency mining.
Some of them reached out to support and got their issue resolved but some of them were not that lucky.

So, I believe this is the issue that caused the testnet-1 to get halted. This is just a theory but I've heard from at least 10 people, some of them are my friends some of them are Kira community members, that they have been banned at the same time on Google Cloud Provider. Their only similarity was that they all have been running Kira Network Testnets.

I believe if you look at the logs you will see that, so many validators got cut off at the same time frame and this might be the reason consensus or network has halted. You can ask the community about the Google banning people for Kira node, you'll probably see more than 30 people.

Just my 2 cents on the situation, wanted to share with y'all. Thanks and looking forward to the Testnet-2!

[denisanix]

probably you are right -- based on the below logs, some validators were caught in shambles
++ jq -rc .consensus_stopped
++ echo {"height":"49999","round":0,"step":"RoundStepCommit","start_time":"2021-04-02T09:52:43.356426452Z","commit_time":"2021-04-02T09:52:46.050147418Z","triggered_timeout_precommit":false,"proposer":"kira10flmchgzq90elwmqpgt5fnp6cfgsctte0vwpsa","precommits":["kira1ekwqt3l2tgasj7p2dyhpd3yfy43f3zke9tqt2v"],"prevotes":[],"noncommits":["kira12ycnw3djqnj347q9h5ppf8sswjlrc5n3pg4nl2","kira1ux58h85fh75se3np26zscsu4wle7h0x29nuesq","kira1kdkmg0exjhdwyndydwadf9c7apsqrxhf9n7jrw"],"average_block_time":7.935450048927537,"consensus_stopped":false}

• CONSENSUS='{\height":"49999","round":0,"step":"RoundStepCommit","start_time":"2021-04-02T09:52:43.356426452Z","commit_time":"2021-04-02T09:52:46.050147418Z","triggered_timeout_precommit":false,"proposer":"kira10flmchgzq90elwmqpgt5fnp6cfgsctte0vwpsa","precommits":["kira1ekwqt3l2tgasj7p2dyhpd3yfy43f3zke9tqt2v"],"prevotes":[],"noncommits":["kira12ycnw3djqnj347q9h5ppf8sswjlrc5n3pg4nl2","kira1ux58h85fh75se3np26zscsu4wle7h0x29nuesq","kira1kdkmg0exjhdwyndydwadf9c7apsqrxhf9n7jrw"],"average_block_time":7.935450048927537,"consensus_stopped":false}

[alkadeta]

Yes! thank you. I think we found the issue! I'm questioning some community members currently and so many of them got banned this morning. I believe my theory is correct 👌

[Creamers158]

Strange thing, cause the precommits on the last block seemed to be there:
"precommits_bit_array": "BA{69:xxx_xxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxx_xxxx} 65/69 = 0.94"

[reb0rn21]

This is my log from validator when the issue occurred:

2021-04-02T09:52:46.274509222Z �[90m9:52AM�[0m �[1m�[31mERR�[0m�[0m Error on ApplyBlock �[36merr=�[0m"commit failed for application: error changing validator set: duplicate entry Validator{EBAA8A1AA4EC636AE3345436C291B51977B06D5B PubKeyEd25519{6DDCB539FEC02B209DC7FAB0F2670CF3FEA7564BE58181A495013D5BB6B7427B} VP:1 A:0} in [Validator{27CD156FF3493A5AF88A9C626EADE6C1B34082CF PubKeyEd25519{10BBF3CAE594EE09C0C5DC14F9C812557BBBEBFAA42446CF9EFBD6DABC0AE3B1} VP:1 A:0} Validator{31140F76FC42E1A7A1711ACE5D268FE2A73DDE81 PubKeyEd25519{F7147BF2372E764A27642D698C81CBE7B6D364A1FE704BB3A63BF08FFC175D7D} VP:1 A:0} Validator{492FFC1E95C6ECD94FA45C4936934D286445056A PubKeyEd25519{539715FAD6BBD4FADE49AC70DFA424E251EAD06CBC85FC07C79C4423823AB2C7} VP:1 A:0} Validator{5729EFBB585174D35B2651423053F71396A8AAD5 PubKeyEd25519{A549CB832960BEABFC42A8DD5A6660CB3B4C4BF5D96E330B2F5D8CD886695DD3} VP:1 A:0} Validator{69471AA0DBC1B27405F112E26A2AE9E34A2951B2 PubKeyEd25519{B7A316A33B885438DD828A82CB53CF3AB52DCFC5F4F78AF77EEE79B7A2B7DBF8} VP:1 A:0} Validator{7AEF92A8D5BEB560F541B386CF659504D5386EB1 PubKeyEd25519{B7AFB13FC0E6A0D9FCC0B15141FDF5C0D66BBE1E3C30D5FA659B4631F266825F} VP:1 A:0} Validator{8482AB39BDA9AF0CB2AADD8DDD382BDDE1800F17 PubKeyEd25519{85A78754C5E9DEC4CE8DD65B79205D0757BF7C885508F03AFE16A04514E7DB65} VP:1 A:0} Validator{D4EC26408087B77E8D307DAE9EE45F080A6152D5 PubKeyEd25519{D0AC6AE1081AD29C64E90F347AE783FCD14857FCE25083860CF8E29ED4649250} VP:1 A:0} Validator{EBAA8A1AA4EC636AE3345436C291B51977B06D5B PubKeyEd25519{6DDCB539FEC02B209DC7FAB0F2670CF3FEA7564BE58181A495013D5BB6B7427B} VP:0 A:0} Validator{EBAA8A1AA4EC636AE3345436C291B51977B06D5B PubKeyEd25519{6DDCB539FEC02B209DC7FAB0F2670CF3FEA7564BE58181A495013D5BB6B7427B} VP:1 A:0} Validator{EE60368B7AE1EC97D27AF22D4F4E03D2F4B77726 PubKeyEd25519{680CEEE5567D9DF73245A3D547AB622D835767FA8C301C0F26580901C512740D} VP:1 A:0} Validator{F7D045ED1B03FC63B3C3E665B93E926BC98507AF PubKeyEd25519{60CA6836A2F01CD79EF469A8ED9618DFA1E70E11B477D88107C2820C5C0C6899} VP:1 A:0}]" �[36mmodule=�[0mconsensus 2021-04-02T09:52:46.287181907Z �[90m9:52AM�[0m �[32mINF�[0m Added to precommit �[36mmodule=�[0mconsensus �[36mprecommits=�[0m"VoteSet{H:49999 R:0 T:SIGNED_MSG_TYPE_PRECOMMIT +2/3:B7A18ABEFE638D58591FD5C3DF2091D937DD4BD067A58E9212EBB40412A5444D:1:D6AB9BCA62E6(0.6956521739130435) BA{69:xxx_x_xx_x__xx_xxxxx_xxxxxxxxxx__x_xxx__x_x_xxxxxxxx_xxxxxx__xx____xx} map[]}" �[36mvote=�[0m{"block_id":{"hash":"B7A18ABEFE638D58591FD5C3DF2091D937DD4BD067A58E9212EBB40412A5444D","parts":{"hash":"D6AB9BCA62E65863F9B9D09B3B2778C3264C8BF4F743EABA9D40DBEF3E7D4BA2","total":1}},"height":49999,"round":0,"signature":"Tqqu76r0AfjMu7t7cSVNwcpXc1NR+G0mFbxPh6QWvpkcSIIJ3c0jJJB1ttxp03hFgu7nskYZ5OiW5g5p7K8FBg==","timestamp":"2021-04-02T09:52:45.158440737Z","type":2,"validator_address":"D59ECC0F8349A5C23DF6DBCCFFDC3E726359EECE","validator_index":57}

[Creamers158]

error changing validator set: duplicate entry Validator{EBAA8A1AA4EC636AE3345436C291B51977B06D5B

[asmodat]

thank you, resolved on branch testnet-2