"producer double-confirming known range" error when testing failover

[jchung00]

Tag: dawn-v4.2.0.
We have two nodes, same producer key and config, and one of them starts with production paused. They both run on the same machine.
Block producing node 1 start producing when the nodes start. Block producing node 2 syncs fine. We run curl -sL http://127.0.0.1:<node-1-port>/v1/producer/pause and curl -sL http://127.0.0.1:<node-2-port>/v1/producer/resume.
Block producing node 2 begins producing with no problems.
Then, we run curl -sL http://127.0.0.1:<node-2-port>/v1/producer/pause and curl -sL http://127.0.0.1:<node-1-port>/v1/producer/resume.
This causes the producer double-confirming known range assertion exception. We checked that this is where the assertion is defined in EOS.IO:

eos/libraries/chain/block_header_state.cpp

Line 158 in 3b70b57

FC_ASSERT( itr->second < result.block_num - h.confirmed, "producer double-confirming known range" );

We also noticed that block producing node 2 can't resync with the chain even when we run it with the --hard-replay-blockchain option.

[gleehokie]

Thanks for the report, I'll bring this to the devs attention.

[wanderingbort]

I will create a more specific issue for this and reference this issue in it. One of the concepts not ready for fail over is the watermark calculated by the producer_plugin. This watermark is used to determine how many blocks to confirm with each produced block. For the particular scenario you describe we can fix this by scanning incoming blocks for producers who are also local producers. That way your backup node will have accurate watermarks Inaccurate watermarks trick on node into double committing past blocks which is a considered Byzantine failure and is why you cannot easily recover

…

On Sat, May 26, 2018, 2:46 PM Greg Lee ***@***.***> wrote: Thanks for the report, I'll bring this to the devs attention. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3442 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACYR4nEsvGMdedtoJLqZWOaWoHncuVy4ks5t2aMMgaJpZM4UO7Ca> .

[noprom]

Look forward to the fix.

[wanderingbort]

We moved this to 1.1 for a few reasons:

1. while this feature is intended to eventually bloom into support for failover its original purpose was not designed with that in mind and there may be several other harder to find issues with having a hot-standby producer for availability.
2. The watermark calculation is conservative when a process starts. So, if we are simulating a dead node where that original node was restarted this would not have occurred. It is only when we are doing live-to-live-to-live node hand-offs with no process restarting that we would see this bug.

This is not to say it isn't something we need to fix. This will get better in short order. However, I wanted to enumerate some of the reasons why we have slipped it to version 1.1