consensus node stop sync block.

[ZhangTao1596]

Consensus node stop sync data, unless restart.

After restart.

This usually happens when a cn ran for a long term.

[igormcoelho]

NF=6 means it considers all other nodes as faulty...meaning its completely isolated from p2p.
This looks like a p2p issue, perhaps the same issue that eventually yields empty blocks... but this is the worst case, when its actually disconnected.
We need to investigate this.

[ZhangTao1596]

I think maybe this cn far behind block height , it refused all normal consensus messages. So for it, it didn't receive any message from other cn. It thinks nf=6.
Why it stop sync from non-consensus nodes?

[vncoelho]

PR #620 mentions this, it is related to network P2P problems as IGOR said.
Perhaps solving this will also solve the problem of nodes with empty mempools.

[vncoelho]

@shargon, I would prefer to remove the bug flag, because in general the consensus comes back to live again, if you wait.
In this sense, the manually reestart mentioned by @KickSeason is solved with the time.

However, it is still some logic that we need to improve on the P2P and also in the way tasksare being cached.

[yongjiema]

I have the same issue even for non-consensus nodes.

[ZhangTao1596]

NGD is testing #862 and #865 .

[shargon]

Fix here: #904

[vncoelho]

We need that eveyone testing (@KickSeason) run on the last Master-2.x.
In addition, that Testnet and MainNet nodes, as soon as possible, updates to up-to-date version (which will be 2.9.3.

Since 2.9.2 we had important fixes on Priorities Ques of Akka Mailboxes and also Memory Leak on TaskManager.

I believe that these new fixes should enter in another version, let's say 2.9.4.