Nodes stopped at height 25709 and failed to produce blocks

[valdok]

Bug description
All the running nodes stopped at height 25709, failed to produce and mine the next block.
Meanwhile some other nodes succeeded to create empty blocks.

[valdok]

Cause
The problem reproduced because of a flaw in the block construction code, which, under rare conditions, generated a wrong commitment to the UTXO state that should be obtained after the block would be interpreted.
It resulted from the untypical transactions (yet valid), which in turn were caused by improper wallet usage.

Technical details.
Each node keeps a track of the current UTXO set, whereas each UTXO consists of a commitment (EC point), and the Maturity, which is the minimum height at which it becomes spendable. So that the commitment to all the UTXO set depends, among other things, on the maturities.
When the block is constructed - the commitment to the resulting UTXO set is put in the to-be-mined header, and this is to prove the authenticity of the block w.r.t. header.
Additionally, after the block is constructed - the cut-through is applied, which is the removal of redundant inputs/outputs (i.e. UTXOs that are created and spent in the same block).

It was assumed that the final UTXO set (and hence its commitment) should not depend on whether the cut-through was applied or not (i.e. creating and deleting an UTXO yields no effect), and the Node that constructed the block calculated the commitment to the UTXO set straight after interpreting all the transaction that are to be put in the block (which is identical to the block, up to sorting and cut-through).

But this assumption turned-out to be inaccurate.

In the reproduced scenarion there was an UTXO A already presented in a block, whereas in the transaction pool there were both a transaction that spends A, and the transaction that creates A once again.
The Node that constructed the block from transactions consumed (deleted) the old A, and then created the new A. Finally: the Node had the new A.
After cut-through the in/out A's are removed. Hence all the Nodes that interpret the final block will remain with old A.

And, as we said, old and new UTXOs are different because their maturities are different. Hence the whole UTXO state was different.

[valdok]

Resolution:
The code that generates blocks was fixed. It calculates the final UTXO commitment based on the final block (after sort and cut-through), exactly as all the other nodes that would interpret it later.
So that it applies all the transactions, rolls them back, constructs the block (including cut-through), and then - yet again, applies block on itself, evaluates the commitment, and rolls the block back.

Remarks
This scenario is untypical, normally each UTXO is created once, the probability to obtain the same UTXO under normal conditions is negligible.

The reason (most probably) was improper use of wallets, that we discourage strongly.
Most probably the wallet was cloned, i.e. the wallet files (including wallet.db) was manually copied. This may lead to the creation of identical UTXOs by different instances of the wallet in different transactions.

In addition to causing this problem (that was fixed), duplicated UTXOs would create hassles to the user. Existence of multiple identical UTXOs in the blockchain is allowed be design (there are important use-cases for this), but would not be handled properly in the current wallet implementation. Users may see temporarily some of their funds "missing" (multiple identical UTXOs are interpreted as a single UTXO).

P.S. At the moment of writing this (Blockchain height 25879) there are no duplicated UTXOs in the blockchain.