Calamari bug on block assignment

[Luca-Poggi]

Hello Manta Team,
I'm Luca from 🧊 Iceberg Nodes 🧊 an active collator on Calamari Network.
I just discover a critical issue on Calamari block assignment on the round robin algorithm.
🧊 Iceberg Nodes 🧊 collator address: dmyhNFR1qUuA8efaYvpW75qGKrYfzrK8ejygttHojeL4ujzUb
As you can see on rewards tab on subscan (https://calamari.subscan.io/account/dmyhNFR1qUuA8efaYvpW75qGKrYfzrK8ejygttHojeL4ujzUb?tab=reward), the collator performed very well since the start of staking with v3.4.0 but since v3.4.1 it had a very low block production (see also stakekma.com).
During v3.4.0, all blocks where assigned to all collators in a PoW-style configuration, the performances were very good, so I would exclude any issue on hardware. This fact is also confirmed by the collator logs (I can't see proposal for block very often as should be the case).
This is what I found:
Go in Polkadot JS --> "Chain state" --> parachainStaking --> selectedCandidates(), the result is:

[
dmu63DLez715hRyhzdigz6akxS2c9W6RQvrToUWuQ1hntcBwF
dmu7rmwTa35Ec5cnNMpn8EpnFPA727sDtpCQwu9uCo2sfnmg1
dmuaG34aVnxirpMsHXu6Mg7RxNN3cxG74ZyjLVEgvzNqBXm2U
dmuazX1JVi1XSd3g7ifaQQnJpodUfmbJVgqP8LXvgXBnsPGtA
dmuomPgt6hJzKpDcEbz2BNNo9uPFXDvzBk7vnLQx6TLBQG85L
dmuuG83f3JeXBmMp7e3XssJzq7rUAuNgAT3z7HoUPWueqpD1V
dmvFayQJ9S7BgbHE2kmnoVq9UfdbfwHpZ4d1revJfVA6X9dGR
dmvPNCD8YaHusmrdtvpB6HG72BibVSpnbHugT893x4Hw9P186
dmvVY24KwgNwoYnHw5EbC8mTUF9CtZeJzCnSGBawWzaRkNHH4
dmvoKqM8n2PVKyiYhm5VpMMnzMdk1z1WZAYDJEDmSLSqRgrbQ
dmvuGKcNe4VEv1rBVcTFEavAsccciEXcWoEi5iQrdR1NNMD1w
dmvvqrfK5AUYH294zTCCiimJRV7CQDDQyC7RAkd5aZgUn9S6f
dmwM2xeWD2BjmCVYddTjB8QyYsktPsx2gySEZYbpdKFNnSGKd
dmx4WhyUDhAjsMf1mRD55qApjxnqSXcSsmweHgcv8seGkrN4R
dmx7NaUig7rdhwTJcnj6VPFaeou4KsvTqkMTcvHz25LcZtNrT
dmxa3MJczFGT92BUQjwsxguUC2t5qFaDdagfpBQWdGkNPJYQ5
dmxbgDpKK6V3Sayr3jz8MpoUJyxiWese6FtL42RRfZXnWewTD
dmxjZSec4Xj3xz3nBEwSHjQSnRGhvcoB4eRabkiw7pSDuv8fW
dmxvivs72h11DBNyKbeF8KQvcksoZsK9uejLpaWygFHZ2fU9z
dmxyqP33GNwS6mA8bsWHQjeyKJM2eUVeCueER44b254ZCMy23
dmy6WPM2KfD7WBxJYS6UG17GHJVCv8kewiwTr6ciVeXLbBpvf
dmyEgR9K8hsgqt47XYnDDJJMaXgHnPLVjTFW8nfRa2RKoj17U
dmyZopEVaerkgSWWTd4WScPkhQgHTeLfMcHVCkQUyL1gu29c3
dmycXK86XZfJV8CuJWoufuY3wq5mnRwhzQmsfQjDvNypyrUDM
dmyhGCWjejSyze6Hcqx43f8PNR9RWwm4EEobo8HehtBb8W8aU
dmyhGnuox8ny9R1efVsWKxNU2FevMxcPZaB66uEJqJhgC4a1W
dmyhNFR1qUuA8efaYvpW75qGKrYfzrK8ejygttHojeL4ujzUb
dmyxfU1bJM5UR5RWsypKm9KQDkVofm3ifp5gVjzs8uQHUmBZb
dmz1cxDw6nC5impJMZVfDwve5AG2s5AeaqSkZvQnEuqVwLYnL
dmzEUqQGSWsFewzpomYcjhLYkeSAvHYwEoKzG2yXcF8YQoJkL
dmzWDne3MxniVDcF4i2nGkfPZa4pfkWL1AXqgSvWgZmDoTcYw
dmzbLejekGYZmfo5FoSznv5bBik7vGowuLxvzqFs2gZo2kANh
dmzbaFDDoYwXrX7Fa5mT2SfLapMZD8dynXPH4JviFEmMQz9Fu
]

This is the list of all active collators. Because you are using a round robin system as block assignment and due to Aura is skipping a block, you used the option 3 here (make the candidate list odd): #794

Now let's keep a portion of chain for example form block #2245079 to block #2245103:

The process is the following (please consider that CJ Calamari collator is inactive with address: dmyhGCWjejSyze6Hcqx43f8PNR9RWwm4EEobo8HehtBb8W8aU):

So at the end the bug is: the Aura is skipping a slot in the first attempt for block assignment (this is OK for option 3 used before: making candidate list odd), BUT it is NOT skipping a block on second attempt for block assignment (making the option 3 used before: making candidate list odd, unuseful for this). In addition to that, since block skipped by collators is random, we can't be sure about even block distribution.

Hot-fix without code changed: please make in contact with CJ Calamari collator (it was a whitelisted Calamari collator, so Rob should have his contact) and make it running, in order to avoid to skip a block every time.
Obviously a fix in the code must be done asap for sure!

Fix to the code:
There are 2 options:

1. Change the approach used here to another one: Fix Nimbus Slot Skipping #794
   or
2. Make Aura skipping a slot also during 2nd attempt of block assignment

Happy to support the team, by providing logs, answering questions and other requests.
Please fix the issue asap