Create a node for the Kiln testnet #88

zah · 2022-03-10T09:26:09Z

We need to prepare a new node for the long-lived Kiln testnet (testing the upcoming eth1-eth2 merge).

My suggestion is to reuse the current host metal-07.he-eu-hel1.nimbus.prater for this purpose by moving all existing Prater validators on it to metal-06.he-eu-hel1.nimbus.prater.

The freed host needs to run 4 beacon nodes paired with 4 Eth1 instances through the --web3-url parameter. Initially, all Eth1 instances will be based on Geth, but in the future we'll run Geth, Nethermind, Besu and Nimbus-eth1. The beacon nodes will be compiled from the kiln-dev-auth branch and started with the --network=kiln parameter. The Geth instances will be compiled from the kiln-merge-v2 branch and started with the --network=kiln parameter.

The text was updated successfully, but these errors were encountered:

jakubgs · 2022-03-10T11:03:02Z

Then I guess I'll be also moving these special options like open_libp2p_ports or num_threads to metal-05:

infra-nimbus/ansible/group_vars/nimbus.prater.yml

Lines 107 to 111 in f2777f6

    
           'metal-07.he-eu-hel1.nimbus.prater': # 4000 each 
        
             - { branch: 'stable',   start: 28000, end: 31000, build_freq: '*-*-* 11:00:00' } 
        
             - { branch: 'testing',  start: 34000, end: 37000, build_freq: '*-*-* 15:00:00', num_threads: 0, branch_override: 'nim-1.6', nim_commit: 'version-1-6' } 
        
             - { branch: 'unstable', start: 31000, end: 34000, build_freq: '*-*-* 13:00:00', open_libp2p_ports: false } 
        
             - { branch: 'libp2p',   start: 37000, end: 40000, build_freq: '*-*-* 17:00:00' }

@zah @arnetheduck cool with that?

Also move validators from it to `metal-06`. #88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-10T11:35:20Z

I've moved the validators to metal-06 and the special config to metal-05: dd43e135

infra-nimbus/ansible/group_vars/nimbus.prater.yml

Lines 95 to 105 in 0e66b27

    
           'metal-05.he-eu-hel1.nimbus.prater': # 60 each 
        
             - { branch: 'stable',   start: 20164, end: 20224, build_freq: '*-*-* 11:00:00' } 
        
             - { branch: 'testing',  start: 20284, end: 20344, build_freq: '*-*-* 15:00:00', num_threads: 0, branch_override: 'nim-1.6', nim_commit: 'version-1-6' } 
        
             - { branch: 'unstable', start: 20224, end: 20284, build_freq: '*-*-* 13:00:00', open_libp2p_ports: false } 
        
             - { branch: 'libp2p',   start: 20344, end: 20404, build_freq: '*-*-* 17:00:00' } 
        
           'metal-06.he-eu-hel1.nimbus.prater': # 4899 each 
        
             - { branch: 'stable',   start: 20404, end: 25303, build_freq: '*-*-* 11:00:00' } 
        
             - { branch: 'testing',  start: 25303, end: 30202, build_freq: '*-*-* 15:00:00', open_libp2p_ports: false, branch_override: 'nim-1.6', nim_commit: 'version-1-6' } 
        
             - { branch: 'unstable', start: 30202, end: 35101, build_freq: '*-*-* 13:00:00' } 
        
             - { branch: 'libp2p',   start: 35101, end: 40000, build_freq: '*-*-* 17:00:00' }

Will be running `wakubridge` to connect with `eth.*` fleets. status-im/infra-nimbus#88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-10T17:08:02Z

I don't see such go-ethereum branch as kiln-merge-v2:
https://github.com/ethereum/go-ethereum/commits/kiln-merge-v2

Or any other including the word kiln for that matter:

jakubgs · 2022-03-10T17:11:13Z

But I see the nimbus-eth2 branch already exists, so I can at least get that built and set up, if without Geth:
https://github.com/status-im/nimbus-eth2/commits/kiln-dev-auth

Also move validators from it to `metal-06`. #88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-11T09:25:42Z

Still no kiln-merge-v2 in go-ethereum repo. For now I'll set up the beacon nodes.

tersec · 2022-03-11T09:45:40Z

It's the one pointed to by https://github.com/status-im/nimbus-eth2/blob/unstable/docs/kiln_geth_local_testnet.md - https://github.com/MariusVanDerWijden/go-ethereum/tree/merge-kiln-v2

jakubgs · 2022-03-11T09:50:36Z

Nice, thanks.

jakubgs · 2022-03-11T10:12:24Z

I've built a Docker image from the merge-kiln-v2 branch:

 > d push statusteam/go-client:merge-kiln-v2
The push refers to repository [docker.io/statusteam/go-client]
9d586a8b231b: Pushed 
6af987bd7441: Pushed 
8d3ac3489996: Mounted from library/alpine 
merge-kiln-v2: digest: sha256:ff9b1d2959eea8e628287288492f9bfc27199efd4069d7bf3216e24da32c1c2b size: 950

Though I assume I will also have to call geth init with a genesis JSON file, which I assume is this:
https://config.kiln.themerge.dev/el/genesis/geth.json

Though not sure what this file is for:
https://config.kiln.themerge.dev/el/genesis/chainspec.json

- #88 - https://config.kiln.themerge.dev/el/bootnodes - https://config.kiln.themerge.dev/el/genesis/geth.json - https://kiln.themerge.dev/ Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-11T11:28:09Z

Ok, here's the setup for four Geth nodes with the Kiln genesis config: 69469913

admin@metal-01.he-eu-hel1.nimbus.kiln:/docker % d
CONTAINER ID   NAMES               IMAGE                                CREATED         STATUS
81345ac09135   geth-kiln-04-node   statusteam/go-client:merge-kiln-v2   2 minutes ago   Up 2 minutes
0d12317db5a2   geth-kiln-03-node   statusteam/go-client:merge-kiln-v2   2 minutes ago   Up 2 minutes
6b954c0cd014   geth-kiln-02-node   statusteam/go-client:merge-kiln-v2   3 minutes ago   Up 3 minutes
d743d7735875   geth-kiln-01-node   statusteam/go-client:merge-kiln-v2   4 minutes ago   Up 4 minutes

admin@metal-01.he-eu-hel1.nimbus.kiln:/docker % for dir in $(echo geth-*); do $dir/rpc.sh eth_syncing | jq .result; done
false
false
false
false

Had to add support for network IDs to Geth role: status-im/infra-role-geth@9647eaea

- #88 - https://config.kiln.themerge.dev/el/bootnodes - https://config.kiln.themerge.dev/el/genesis/geth.json - https://kiln.themerge.dev/ Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-11T12:07:28Z

run 4 beacon nodes paired with 4 Eth1 instances through the --web3-url parameter

Just to be clea @zah, that means 1-to-1 connections, meaning 01 Geth node to 01 Beacon node?

jakubgs · 2022-03-11T12:18:32Z

I also remember that for Nocturne network we needed consensus API enabled. I see in this script that they add engine, so I added that. Is there anything else we need there?

jakubgs · 2022-03-11T12:38:33Z

This is interesting:

{"lvl":"FAT","ts":"2022-03-11 12:29:30.613+00:00","msg":"config.yaml not found for network","networkName":"kiln"}

zah · 2022-03-11T12:46:46Z

Just to be clea @zah, that means 1-to-1 connections, meaning 01 Geth node to 01 Beacon node?

Yes

jakubgs · 2022-03-11T12:53:20Z

Is the config the error referring to this one?
https://config.kiln.themerge.dev/cl/genesis/config.yaml

jakubgs · 2022-03-11T13:02:41Z

Apparently the --network flag supports passing a path to the folder with network config, like this one:
https://github.com/eth-clients/merge-testnets/tree/main/kiln

Though the help message says nothing about such capability:

     --network                 The Eth2 network to join [=mainnet].

jakubgs · 2022-03-11T13:07:49Z

I cloned the repo to /data and then symlinked the kiln folder in the service path:

admin@metal-01.he-eu-hel1.nimbus.kiln:/data/beacon-node-kiln-devauth-01 % ll
total 24K
-rwxr-x---  1 nimbus staff  602 Mar 11 12:22 ansible_enabled.sh
-rwxrwxr-x  1 nimbus staff 2.2K Mar 11 12:24 build.sh
drwxrwx---  3 nimbus staff 4.0K Mar 11 13:05 data
lrwxrwxrwx  1 root   root    22 Mar 11 13:04 kiln -> ../merge-testnets/kiln
lrwxrwxrwx  1 root   root    44 Mar 11 12:29 logs -> /var/log/service/beacon-node-kiln-devauth-01
drwxr-xr-x 21 nimbus staff 4.0K Mar 11 12:26 repo
-rwxr-x---  1 nimbus staff 1.1K Mar 11 12:22 restart.sh
-rwxr-x---  1 nimbus staff  649 Mar 11 12:22 rpc.sh

And because the service path is used as CWD for the service it appears to work:

admin@metal-01.he-eu-hel1.nimbus.kiln:/data/beacon-node-kiln-devauth-01 % tail -n1 logs/service.log 
{"lvl":"DBG","ts":"2022-03-11 13:05:12.411+00:00","msg":"Synchronization loop sleeping","topics":"beacnde","wall_head_slot":0,"local_head_slot":0,"difference":0,"max_head_age":8,"direction":"Forward"}

So both relative paths to CWD and symlinks work.

jakubgs · 2022-03-11T13:21:32Z

Looks correct... maybe:

admin@metal-01.he-eu-hel1.nimbus.kiln:~ % for i in $(seq 0 6); do curl -s localhost:930$i/eth/v1/node/syncing; echo; done   
{"data":{"head_slot":"0","sync_distance":"0","is_syncing":false}}
{"data":{"head_slot":"0","sync_distance":"0","is_syncing":false}}
{"data":{"head_slot":"0","sync_distance":"0","is_syncing":false}}
{"data":{"head_slot":"0","sync_distance":"0","is_syncing":false}}

admin@metal-01.he-eu-hel1.nimbus.kiln:~ % for i in $(seq 9301 9304); do c localhost:$i/eth/v1/node/peer_count; echo; done    
{"data":{"disconnected":"0","connecting":"0","connected":"17","disconnecting":"0"}}
{"data":{"disconnected":"1","connecting":"0","connected":"16","disconnecting":"0"}}
{"data":{"disconnected":"1","connecting":"0","connected":"14","disconnecting":"0"}}
{"data":{"disconnected":"1","connecting":"0","connected":"10","disconnecting":"0"}}

jakubgs · 2022-03-11T13:26:07Z

I think this is correct:

admin@metal-01.he-eu-hel1.nimbus.kiln:~ % for i in $(seq 9301 9304); do c localhost:$i/eth/v1/config/spec | jq .data.MIN_GENESIS_TIME; done
"1647007200"
"1647007200"
"1647007200"
"1647007200"

# Mar 11th, 2022, 14:00 UTC
MIN_GENESIS_TIME: 1647007200

https://github.com/eth-clients/merge-testnets/blob/f10dde7753b50291c257246f3d303dfdfb3214ef/kiln/config.yaml#L6-L7

jakubgs · 2022-03-11T13:29:20Z

Some extra bootnodes added in: eth-clients/merge-testnets#6
Added in: a4e54f2c

jakubgs · 2022-03-11T14:37:33Z

We tried generating the validators together with @tersec but we incorrectly thought the tools to do that are sub-commands of the nimbus_beacon_node, since this doc appeared to be outdated:
https://nimbus.guide/more-keys.html

Initially I thought the right sub-command is deposits createTestnetDeposits, but that's not what we need.

Turns out the deposit command is a separate binary available from this repo:
https://github.com/ethereum/eth2.0-deposit-cli/

But we already missed genesis.

#88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs · 2022-03-11T14:41:24Z

We had beacon nodes deployed 30 minutes before genesis: 0d5ef90c

admin@metal-01.he-eu-hel1.nimbus.kiln:/data % s list-units 'beacon-node-*'
  UNIT                                LOAD      ACTIVE SUB     DESCRIPTION                                       
● beacon-node-kiln-devauth-00.service not-found failed failed  beacon-node-kiln-devauth-00.service               
  beacon-node-kiln-devauth-01.service loaded    active running Nimbus Beacon Node on kiln network (kiln-dev-auth)
  beacon-node-kiln-devauth-02.service loaded    active running Nimbus Beacon Node on kiln network (kiln-dev-auth)
  beacon-node-kiln-devauth-03.service loaded    active running Nimbus Beacon Node on kiln network (kiln-dev-auth)
  beacon-node-kiln-devauth-04.service loaded    active running Nimbus Beacon Node on kiln network (kiln-dev-auth)

But we didn't get the validators generated on time.

jakubgs · 2022-03-11T15:46:49Z

Looks like @zah changes in status-im/nimbus-eth2@bf127920 broke starting of the node with --network=kiln:

{"lvl":"FAT","ts":"2022-03-11 15:37:21.396+00:00","msg":"No database and no genesis snapshot found: supply a genesis.ssz with the network configuration, or compile the beacon node with the -d:has_genesis_detection option in order to support monitoring for genesis events","topics":"beacnde"}

jakubgs · 2022-03-11T15:59:49Z

Added and deployed deposits layout of 125/500 per node: 38b16053

root@metal-01 ~ # ls /data/beacon-node-kiln-devauth-*/data/shared_kiln_0/secrets/ | wc -l
500

Had to pin nimbus-eth2 repo at status-im/nimbus-eth2@a35c5f8 to avoid startup issues.

jakubgs · 2022-03-11T18:19:24Z

Also enabled --proposer-boosting-debug: d33ec703

 > a nimbus.kiln --become -a 'grep proposer-boosting-debug /etc/systemd/system/beacon-node-kiln-devauth-*.service'
metal-01.he-eu-hel1.nimbus.kiln | CHANGED | rc=0 >>
/etc/systemd/system/beacon-node-kiln-devauth-01.service:    --proposer-boosting-debug=true \
/etc/systemd/system/beacon-node-kiln-devauth-02.service:    --proposer-boosting-debug=true \
/etc/systemd/system/beacon-node-kiln-devauth-03.service:    --proposer-boosting-debug=true \
/etc/systemd/system/beacon-node-kiln-devauth-04.service:    --proposer-boosting-debug=true \

jakubgs · 2022-03-14T08:37:18Z

I see no issues with the nodes, so I'm closing this as completed.

jakubgs added a commit that referenced this issue Mar 10, 2022

move metal-07.he-eu-hel1.nimbus.prater to nimbus.kiln

0e66b27

Also move validators from it to `metal-06`. #88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs added a commit to status-im/infra-status-legacy that referenced this issue Mar 10, 2022

deploy a new bridge host in each fleet

ac96e85

Will be running `wakubridge` to connect with `eth.*` fleets. status-im/infra-nimbus#88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs added a commit that referenced this issue Mar 10, 2022

move metal-07.he-eu-hel1.nimbus.prater to nimbus.kiln

dd43e13

Also move validators from it to `metal-06`. #88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs added a commit that referenced this issue Mar 11, 2022

nimbus.kiln: configure beacon nodes without validators

0d5ef90

#88 Signed-off-by: Jakub Sokołowski <jakub@status.im>

jakubgs closed this as completed Mar 14, 2022

jakubgs self-assigned this Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a node for the Kiln testnet #88

Create a node for the Kiln testnet #88

zah commented Mar 10, 2022 •

edited

Loading

jakubgs commented Mar 10, 2022

jakubgs commented Mar 10, 2022 •

edited

Loading

jakubgs commented Mar 10, 2022

jakubgs commented Mar 10, 2022

jakubgs commented Mar 11, 2022

tersec commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

zah commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 14, 2022

Create a node for the Kiln testnet #88

Create a node for the Kiln testnet #88

Comments

zah commented Mar 10, 2022 • edited Loading

jakubgs commented Mar 10, 2022

jakubgs commented Mar 10, 2022 • edited Loading

jakubgs commented Mar 10, 2022

jakubgs commented Mar 10, 2022

jakubgs commented Mar 11, 2022

tersec commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 • edited Loading

jakubgs commented Mar 11, 2022 • edited Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

zah commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 • edited Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022 • edited Loading

jakubgs commented Mar 11, 2022 • edited Loading

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 11, 2022

jakubgs commented Mar 14, 2022

zah commented Mar 10, 2022 •

edited

Loading

jakubgs commented Mar 10, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading

jakubgs commented Mar 11, 2022 •

edited

Loading