Rotate external validators for non-beacon shards.

[Frozen]

When the feature NonBeaconLeaderRotation is activated, the leaders will be rotated on shards 1-3 an equal amount of blocks depending on the epoch length and slots count. Shard 0 will rotate only the internal validators.

These are all outdated.

[sophoah]

The PR looks ok. My only concern right now is the simple approach for the leader rotation that doesn't take into consideration performance of the validator. (ie it could be an issue if a big validator keep missing signature in a row and the next bls key is still that validator)

next would be to perform proper testing. Localnet by default doesn't have external validator (and leader) so we need to customize the localnet for that particular test.

[Frozen]

When feature activated on epoch 5, it starts rotate. Minimum 3 blocks. This is from blocks that were inserted:

epoch | block | leader pub key (part)

4 35 65f55eb3052f9e9f632b2923be
4 36 65f55eb3052f9e9f632b2923be
4 37 65f55eb3052f9e9f632b2923be
4 38 65f55eb3052f9e9f632b2923be
4 39 65f55eb3052f9e9f632b2923be
4 40 65f55eb3052f9e9f632b2923be
4 41 65f55eb3052f9e9f632b2923be
4 42 65f55eb3052f9e9f632b2923be
4 43 65f55eb3052f9e9f632b2923be
4 44 65f55eb3052f9e9f632b2923be
5 45 65f55eb3052f9e9f632b2923be
5 46 65f55eb3052f9e9f632b2923be
5 47 65f55eb3052f9e9f632b2923be
5 48 02c8ff0b88f313717bc3a627d2
5 49 02c8ff0b88f313717bc3a627d2
5 50 02c8ff0b88f313717bc3a627d2
5 51 e751ec995defe4931273aaebcb
5 52 e751ec995defe4931273aaebcb
5 53 e751ec995defe4931273aaebcb
5 54 2d61379e44a772e5757e27ee2b
6 55 65f55eb3052f9e9f632b2923be
6 56 65f55eb3052f9e9f632b2923be
6 57 65f55eb3052f9e9f632b2923be
6 58 02c8ff0b88f313717bc3a627d2
6 59 02c8ff0b88f313717bc3a627d2
6 60 02c8ff0b88f313717bc3a627d2
6 61 e751ec995defe4931273aaebcb
6 62 e751ec995defe4931273aaebcb
6 63 e751ec995defe4931273aaebcb
6 64 2d61379e44a772e5757e27ee2b
7 65 65f55eb3052f9e9f632b2923be
7 66 65f55eb3052f9e9f632b2923be
7 67 65f55eb3052f9e9f632b2923be

[sophoah]

@Frozen as mentioned over chat, the current localnet with make debug is only testing with internal bls keys already hardcoded in the codebase. Feel free to use use #4388 to build a localnet with external validator.

The test I performed showed a lower number of blocks per epoch for shard 1 vs shard 0. Below is shard 1 validator with 2 keys. with 64 blocks per epoch, there should be 128 signatures like it had in epoch 4. However at activation of leader rotation activation feature in epoch 5 for non beacon chain, then epoch 6 leader rotation for epoch chain, signature goes down significantly lower

        {
          "blocks": {
            "signed": 30,
            "to-sign": 30
          },
          "epoch": 8
        },
        {
          "blocks": {
            "signed": 30,
            "to-sign": 30
          },
          "epoch": 7
        },
        {
          "blocks": {
            "signed": 38,
            "to-sign": 40
          },
          "epoch": 6
        },
        {
          "blocks": {
            "signed": 108,
            "to-sign": 112
          },
          "epoch": 5
        },
        {
          "blocks": {
            "signed": 128,
            "to-sign": 128
          },
          "epoch": 4
        },

See the same for shard 0 which doesn't any drop in number of blocks signed but noticed that the performance of the validator is not very good, we should have all the time 128 out of 128

        {
          "blocks": {
            "signed": 127,
            "to-sign": 128
          },
          "epoch": 9
        },
        {
          "blocks": {
            "signed": 120,
            "to-sign": 128
          },
          "epoch": 8
        },
        {
          "blocks": {
            "signed": 119,
            "to-sign": 128
          },
          "epoch": 7
        },
        {
          "blocks": {
            "signed": 124,
            "to-sign": 128
          },
          "epoch": 6
        },
        {
          "blocks": {
            "signed": 122,
            "to-sign": 128
          },
          "epoch": 5
        },
        {
          "blocks": {
            "signed": 128,
            "to-sign": 128
          },
          "epoch": 4
        },

[sophoah]

I figured another issue here with this PR which ofc will only happen in localnet or testnet. When there is no external validator at the staking epoch activation feature, the network will stop.

[sophoah]

the above issue is fixed by adding another FN node in the localnet (had one, and we needed two)

[sophoah]

@Frozen could you rebase this PR with main there are some conflicts when I am trying to do locally.

[sophoah]

@Frozen after rebasing the PR could you test localnet with external.
localnet with external node can be started with

bash test/debug-external.sh

once started and in another console the validator are created with

bash test/build-localnet-validator.sh

There are a total of 6 external BLS key (3 s0 and 3 s1) and 2 validator created in each shard. First one with 2 keys, and the second one with one key.

check the validator statistics

# s0 validator with 2 s0 keys
hmy blockchain validator information one17ughrllgnzx9sfa46p568k8rdmtz7qj85slc6t --node="http://localhost:9500"
# s0 validator with 1 s0 keys
hmy blockchain validator information one1auqndgthqu5lznsn7tuma8s5333cq0y07cwc6x --node="http://localhost:9500"
# s1 validator with 2 s1 keys
hmy blockchain validator information one19aw2wcr5y4lxeuwt0ajgt5aw3a3qkjdgg67ygj --node="http://localhost:9500"
# s1 validator with 1 s1 keys
hmy blockchain validator information one1eenp9ujcrmyaq22ef6jrpry2k97tjz4xs6ppcf --node="http://localhost:9500"

the logs of those validator can be accessed in
tmp-log/logs-yyyymmdd-xxxxx

ie

 ls tmp_log/log-20230329-092724/
bootnode.log                           zerolog-validator-127.0.0.1-9006.log   zerolog-validator-127.0.0.1-9052.log   zerolog-validator-127.0.0.1-9108.log
r.log                                  zerolog-validator-127.0.0.1-9008.log   zerolog-validator-127.0.0.1-9096.log   zerolog-validator-127.0.0.1-9110.log
zerolog-validator-#127.0.0.1-9054.log  zerolog-validator-127.0.0.1-9010.log   zerolog-validator-127.0.0.1-9098.log   zerolog-validator-127.0.0.1-9114.log
zerolog-validator-#127.0.0.1-9154.log  zerolog-validator-127.0.0.1-9014.log   zerolog-validator-127.0.0.1-9100.log   zerolog-validator-127.0.0.1-9116.log
zerolog-validator-127.0.0.1-9000.log   zerolog-validator-127.0.0.1-9016.log   zerolog-validator-127.0.0.1-9102.log   zerolog-validator-127.0.0.1-9118.log
zerolog-validator-127.0.0.1-9002.log   zerolog-validator-127.0.0.1-9018.log   zerolog-validator-127.0.0.1-9104.log   zerolog-validator-127.0.0.1-9150.log
zerolog-validator-127.0.0.1-9004.log   zerolog-validator-127.0.0.1-9050.log   zerolog-validator-127.0.0.1-9106.log   zerolog-validator-127.0.0.1-9152.log

and you can match the node logs with the port configured for the given node in https://github.com/harmony-one/harmony/blob/main/test/configs/local-resharding-with-external.txt

i can't manage to have stable localnet right now, it would eventually fails with at epoch change with validator not able to sign all the time

[ONECasey]

Lets get this in dev but lets also open up the leader rotation for the beacon chain.

[ONECasey]

@Frozen Can we also enable slashing?