Incorrect (really big) number of confirmed blocks in header when producer get voted out then voted in again

[taokayan]

We recently find a bug in producer plugin that cause the synchronization fails.

This is the short story:

1. producer kayan1111132 produced block at #145488
2. kayan1111132 no longer produce block for pretty long time because he was voted out.
3. kayan1111132 got voted in again and produced block at #228494, it tried to confirm the range from #145489 to #228493 by the producer plugin, but due to the type of uint16_t, the resulting number will be wrapped into 17469
4. all the validating nodes noticed that kayan1111132 is the new producer, and set his last produced block number to be the last irreversible number (#228000), which means that kayan1111132 should only confirm block range from #228001 to #228493. This results in double-confirm exception

In my understanding, we should fix this in the producer plugin, by not confirming the blocks that already became irreversible, instead of just checking the watermark (last produce block num).

related log:

2315501ms thread-0   producer_plugin.cpp:987       produce_block        ] Produced block 00023850866b7827... #145488 @ 2018-06-01T16:38:35.500 signed by kayan1111132 [trxs: 0, lib: 145128, confirmed: 0]
...
...
407502ms thread-0   controller.cpp:752            start_block          ] promoting proposed schedule (set in block 227656) to pending; current block: 227989 lib: 227664 schedule: {"version":17,"producers":[{"produce
r_name":"dhanesh","block_signing_key":"EOS5jKcVPZgsog1d2k4DFjay4Jdjr1rkpFbpBtKUShNTy3gQKuSJs"},{"producer_name":"kayan","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"
kayan1111111","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111112","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name
":"kayan1111113","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111114","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_n
ame":"kayan1111115","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111121","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"produce
r_name":"kayan1111122","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111123","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"prod
ucer_name":"kayan1111124","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111125","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"p
roducer_name":"kayan1111131","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111132","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},
{"producer_name":"kayan1111133","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111134","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w
"},{"producer_name":"kayan1111135","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111141","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkM
p1w"},{"producer_name":"kayan1111142","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJNkMp1w"},{"producer_name":"kayan1111143","block_signing_key":"EOS5d9ezziQCNU5cnrTPkLxCMfqmbSg1Y5ZfosKrrwAj1YJ
NkMp1w"},{"producer_name":"thezealot","block_signing_key":"EOS8dGGh6NXTwPm81dYH94hjW5QjQnEUaZ9uEFF83RErcUMvfCiqn"}]} 

...

671501ms thread-0   producer_plugin.cpp:987       produce_block        ] Produced block 00037c8d504d7213... #228493 @ 2018-06-02T04:11:11.500 signed by kayan1111131 [trxs: 0, lib: 228000, confirmed: 0]
672003ms thread-0   producer_plugin.cpp:987       produce_block        ] Produced block 00037c8ee816b731... #228494 @ 2018-06-02T04:11:12.000 signed by kayan1111132 [trxs: 0, lib: 228180, confirmed: 17469]

[spartucus]

Great ~

just checking the watermark (last produce block num)

So last produce block num - 1 means irreversible?

[taokayan]

Later I found out this is not really a big issue.
in the reality, if this happens, at most one block will not be accepted by all other nodes, and the producer of that block will then abandon the invalid block and switch back to the main(longer) chain.

[Jovonni]

Where can I find the actual content of a block on EOS?