[documentation] instruction on how to migrate the non-validating nodes

[LeoHChen]

Problem/limitation At Hand

Since we have a new database schema for the explorer node db, all the non-validating nodes need to migrate to the new DB schema.
This is needed to resolve the explorer node catch-up issue. #3740

We need to provide clear instructions to all the exchange partners that run the non-validating node as the RPC endpoint for the db migration.

[LeoHChen]

Please work with @JackyWYX on the instructions. Basically, we need to create a rclone snapshot of an existing new explorer DB and then ask the exchange partners to rclone the new explorer DB to replace their old explorer DB after the binary upgrade.

[LeoHChen]

@yelllowsin can help out on the documentation part.

[sophoah]

Assumption

• The p2p port is default at 9000
• The database folder (ie harmony_db and explorer_storage) are in the same directory as the home harmony node binary
• Harmony node has been installed with systemd

new disk recommendation

Current DB Is at 7.8 TB, Harmony is is now using i3en.6xlarge which has 2 x 7,5 TB configured in raid 0, it is strongly recommended to use a local NVMe disk

Download the new explorer DB

wget https://s3.us-west-1.amazonaws.com/pub.harmony.one/mainnet.archival/harmony_storage_db_0/explorer_storage_s0.tar.gz

Perform the upgrade as usual without restarting the node

mkdir -p backup
cp harmony backup
curl -LO https://harmony.one/binary && mv binary harmony && chmod +x harmony
sudo service harmony stop

Backing up the old explorer DB

mv explorer_storage_0.0.0.0_9000 explorer_storage_0.0.0.0_9000.backup

Extracting the new DB

tar xvzf explorer_storage_s0.tar.gz

Check your RPC rate limit and increase accordingly

Check

curl -s http://localhost:9900/metrics | grep ^hmy_rpc

hmy_rpc_over_ratelimit{rate_limit="50000"} 1721

Change the rate limit

update harmony.conf in the section [RPCOpt] the RequestsPerSecond from default 1000 to 50000

[RPCOpt]
  RequestsPerSecond = 50000

start the harmony node

sudo service harmony start

[LeoHChen]

Do not forget to add instructions to increase the RPC rate limiter. The default rate limit is 1000. I suggest partners set it to at least 50000.

Use

curl http://localhost:9900/metrics | grep ^hmy_rpc

To check if the rate limiter is reached and set to a higher limit accordingly.

Also, please advise all exchange partners to use i3en.6xlarge for archival explorer nodes or equivalent VPS, and use the local NVMe drive to handle disk IO.

[JB273]

@LeoHChen @sophoah How will we address the needed space issue for the future for this nodes, especially the ones for Shard 0!?
7.5TB is really a lot, it's almost the same size as the ETH archival node.

[yelllowsin]

@LeoHChen @sophoah I added the upgrade instructions here:https://docs.harmony.one/home/network/validators/node-setup/installing-updating/upgrading-node/using-binary#non-validating-explorer-nodes

And requirements here (they are linked each to the other): https://docs.harmony.one/home/network/validators/server-setup/requirements#explorer-node-recommendation

Let me know if you have further changes I should make.

[yelllowsin]

@LeoHChen One more question, how do we address syncing for shards other than 0?

I can instruct users to download explorer_storage_s0.tar.gz, but what about other shards? Also, is there a process that updates these snapshots from times to times. Currently we are using Rclone for new installations.

[JackyWYX]

@LeoHChen One more question, how do we address syncing for shards other than 0?

I can instruct users to download explorer_storage_s0.tar.gz, but what about other shards? Also, is there a process that updates these snapshots from times to times. Currently we are using Rclone for new installations.

Running the new binary will automatically migrate the explorer DB. Since other shards have much less data (about 100~200MB), I think that the migration process itself should be fine, not really need to create a new snapshot for it. So basically, there are no extra steps needed for shard chain.

[yelllowsin]

@LeoHChen One more question, how do we address syncing for shards other than 0?
I can instruct users to download explorer_storage_s0.tar.gz, but what about other shards? Also, is there a process that updates these snapshots from times to times. Currently we are using Rclone for new installations.

Running the new binary will automatically migrate the explorer DB. Since other shards have much less data (about 100~200MB), I think that the migration process itself should be fine, not really need to create a new snapshot for it. So basically, there are no extra steps needed for shard chain.

Got it, so basically we can remove from the docs the download of explorer_storage_s0.tar.gz? Not sure if I get the point of having it if the binary will do the conversion anyway.

[JackyWYX]

Got it, so basically we can remove from the docs the download of explorer_storage_s0.tar.gz? Not sure if I get the point of having it if the binary will do the conversion anyway.

It is still needed for shard 0 explorer since explorer db for s0 is massive. It takes ~5 hours to migrate and it has been reported there are some dirty data in some explorer node that caused migration to fail. So if a shard 0 explorer node is migrating from older version, it is recommended to directly download the snapshot.

For longer term execution, let's have a discussion of whether we shall add the snapshot of explorer db to the snapshot instance so that the explorer db can be rcloned just as the chain db. I think for new node starting for the first time, it is recommended to download the snapshot instead of generating everything from scratch to make the bootstrap faster.@LeoHChen

[sophoah]

ess the needed space issue for the future for this nodes, especially the ones for Shard 0!?
7.5TB is really a lot, it's almost the same size as the ETH archival node.

There are some work to be done for sure... @rlan35 Any insight ?