-
Notifications
You must be signed in to change notification settings - Fork 5
Deploy OasisEth Paratime on Oasis Mainnet
This document is updated to fit the Oasis Cobalt Upgrade. If you are going to deploy your paratime after the Cobalt Upgrade, please check the details carefully. There are several breaking changes
The Oasis network will upgrade at 2021/04/28. If you are looking for the upgrade document. Please refer to this page
Dear OasisEth Paratime operators, Before you start, please check the following information first:
- The OasisEth Paratime MAINNET network parameters are all set.
- **The genesis token unlock timestamp is set to
2021/03/05 13:00:00 UTC+8
. - Please take 1 minute to fill out this validator info collection form if you have not filled it out. https://forms.gle/HzMqyAnaTJbtoh3G8
- To join the OasisEth Paratime committee, please refer to this announcement for the detail timeline: Join OsisEth as a node to get OETH token reward.
Latest update: 2021/05/20 15:38 UTC+8
Updated history:
- 2021/05/20 15:38 UTC+8, Update the runtime id to fix the issues caused by Oasis Cobalt Upgrade.
- 2021/04/28 16:46 UTC+8, Update document for the Oasis Cobalt Upgrade.
- 2021/04/26 19:00 UTC+8, Add Oasis Cobalt Upgrade information.
- 2021/03/24 12:14 UTC+8, Add a link to joining process.
- 2021/03/16 18:27 UTC+8, move the entity registration part to the first step.
- 2021/03/15 14:49 UTC+8, update software version: OasisEth v1.5.4, Oasis-Core v20.12.5
- 2021/03/14 21:18 UTC+8, add status.oasiseth.org website link.
- 2021/03/13 21:23 UTC+8, update the staked amount requirement since the storage role is disabled for now.
- 2021/03/10 22:39 UTC+8, remove storage role when starting an Oasis paratime node.
- 2021/03/07 00:57 UTC+8, add expected output when an OasisEth paratime node is in the working committee; Add command to get the runtime node list.
- 2021/03/05 17:45 UTC+8, add expected output when an OasisEth paratime node is running normally.
- 2021/03/04 15:47 UTC+8, add a note to mention that the node identity will also be removed when doing the clean up previous node step.
- 2021/03/04 14:42 UTC+8, correct the runtime id.
In this guide, you will learn how to set up an OasisEth Paratime to join our committee. You will deploy two different nodes including
- an Oasis Ethereum ParaTime node (required) and
- a gateway node that provides public web3 RPC services (optional)
We suggest each node be deployed on a separate virtual machine because the network ports will have a conflict with different nodes on the same machine.
The web3 gateway node is entirely optional. Your ParaTime node will participate in the consensus and earn OETHs regardless of whether you run the web3 gateway. However, if you do run a web3 gateway, it must reside on its own virtual machine.
In summary, your node network may look like this:
- Required: an Oasis validator node
- Required: a ParaTime node
- Optional: a web3 gateway
To learn more about the Oasis Ethereum ParaTime, please visit https://www.oasiseth.org/
- NOTICE
- Overview
- TOC
- Minimum Requirements for running a paratime node
- Current MAINNET network parameters
- Recommended Hardware
- If you have run the OasisEth Paratime mainnet beta
- Part 1. Submit your entity registration to join the OasisEth committee
-
Part 2. Setting up a ParaTime node
- 2-1. Environment
- 2-2 Setup a ParaTime Node
- 2-3 Check Your ParaTime node Status
- 2-4 Your paratime nodes are synced and registered
- 2-5 (Optional) Set up the web3 RPC service within the paratime node
- Part 3. (Optional) Set up a gateway node for the web3 JSONRPC
- Maintenance
- Reference
- Troubleshooting
To run the ParaTime node, you MUST already have an Oasis Validator Node with a registered entity ID on the Oasis Mainnet network. Hence, you are probably already running:
- (Required) One validator node with at least 300 ROSEs staked
The 300 ROSEs staked contains:
- (a) 100 ROSEs for the entity registration
- (b) 100 ROSEs for the validator node
- (c) 100 ROSEs for the paratime node with compute role only
According to the paratime architecture, a paratime node contains two roles (a) compute role and (b) storage role.
- (a) compute role will do the execution jobs such as transferring OETH, deploying smart contracts, and executing the smart contracts.
- (b) storage role will handle the paratime states such as smart contract bytecode, account balance... But the Oasis-Core v20.12.x has a bug in the paratime storage node synchronization. As a workaround, we have to disable the storage role for the paratime node operators. So the paratime node operators will only run the paratime node with compute role.
If you are starting from setting up an oasis validator node, please notice that oasis entity registration and validator node registration will ask for extra ROSEs staked. See oasis document for more details: https://docs.oasis.dev/general/contribute-to-the-network/run-validator#stake-requirements According to the oasis document, your first validator node setup should have:
- 100 ROSE staked for your entity's registration since that is the current entity's staking threshold.
- 100 ROSE staked for your validator node's registration since that is the current validator node's staking threshold.
Please notice that you MUST run a validator node with at least 300 ROSEs staked to receive the OETH reward. We only accept those entity IDs who are running a validator node on the Oasis mainnet. And your Oasis validator node doesn't need to be the Oasis consensus committee (the top 80 validators).
The genesis token unlock timestamp is set to 2021/03/05 13:00:00 UTC+8
. If you want to join the OETH reward in the first month, please deploy your paratime node before the following timestamp 2021/03/08 13:00:00 UTC+8
.
-
RUNTIME_ID:
000000000000000000000000000000000000000000000000000000000000ff04
-
OASIS_SEED_NODE_ADDRESS:
E27F6B7A350B4CC2B48A6CBE94B0A02B0DCB0BF3@35.199.49.168:26656
-
CHAIN ID:
69
-
The genesis timestamp is set to
2021/03/05 13:00:00 UTC+8
. -
The coinbase address is set to
0x22aa096E896e16c0B99bdb93D6084C947c3bddAB
. -
SSVM-EVMC Version: evmc6-0.1.1
-
Oasis-SSVM-Runtime & Gateway Version: v1.7.0
-
Oasis-node Version(Oasis-Core-Binary): v21.1.1
-
Oasis-Genesis: mainnet 2021-04-28
- CPU: 2.0 GHz x86-64 CPU
- Memory: 4GB RAM
- Disk: 500GB High-Speed Storage (SSD)
- For your reference, we deploy our own ParaTime node on Azure B2S instances (2 VCPU, 4GB Memory, 500GB Disk space)
Please remove the DATA_DIR
folder to clean up all the node information and runtime data of mainnet beta
. In case that you use the wrong configuration to set up your new OasisEth paratime mainnet node.
Please notice that remove the DATA_DIR
folder will also remove your previous paratime node identity. This is expected. After removing the DATA_DIR
you will also need to update your entity registration with the new paratime node identity. And do the oasis-node registry entity update
step again.
rm -rf /path/to/your/oasis-ssvm-runtime # Remove the previous version of oasis-ssvm-runtime. That is for the mainnet beta.
rm -rf /path/to/your/node-data # It will be the /home/user/mainnet/node in our previous tutorial.
Please refer to our ParaTime entities repo.
Follow the README and an example to learn how to submit your entity id.
After submitting the pull request, you can go ahead to do the remaining steps to deploy your paratime node. However, your paratime node may not be registered before your pull request is merged.
Prepare the environment and download pre-built binaries
Our pre-built binaries only support Ubuntu 18.04 and Ubuntu 20.04
Please use the Ubuntu 18.04/20.04 to set up a machine.
In this step, you will create a folder at path /ssvm
and download ssvm dependency into this folder.
Please notice that the libssvm-evmc.so
MUST be put into /ssvm
folder.
mkdir -p /ssvm \
&& cd /ssvm \
&& curl -LO https://github.com/second-state/ssvm-evmc/releases/download/evmc6-0.1.1-rc1/libssvm-evmc.so
In the following guide, we use /home/user/mainnet
as an example. Feel free to replace this path with your own /path/to/mainnet/folder/
.
mkdir -p /home/user/mainnet
cd /home/user/mainnet
# Oasis-Core
wget <Oasis-Core-Binary URL, see the network parameter on the above section>
tar zxvf oasis_core_21.1.1_linux_amd64.tar.gz
rm oasis_core_21.1.1_linux_amd64.tar.gz
# Mainnet genesis
wget <Oasis-Genesis URL, see the network parameter on the above section>
# oasis-ssvm-runtime
wget <Oasis-SSVM-runtime URL, see the network parameter on the above section>
tar zxvf oasis-ssvm-runtime.tgz
rm oasis-ssvm-runtime.tgz
mkdir -p /home/user/mainnet/entity
cd /home/user/mainnet/entity
scp YourValidatorNode:/path/to/your/entity.json ./entity.json
/ssvm/libssvm-evmc.so
/home/user/mainnet
├── entity
│ └── entity.json
├── genesis.json
├── gateway
├── oasis-node
└── oasis-ssvm-runtime
- Make sure you should use a proper IP for
$EXTERNAL_IP
. - You should expose the following ports:
9100
,9200
,26656
,8545
-
9100
: This port is for the Oasis ParaTime gateway connection. -
9200
: This port is for the Oasis ParaTime P2P connection. -
26656
: This port is for tendermint consensus gossip. -
8545
: This port is for the JSONRPC endpoint.
-
mkdir -m 700 -p /home/user/mainnet/node
# The following commands should be executed in /home/user/mainnet folder
cd /home/user/mainnet
export RUNTIME_ID=<SSVM_RUNTIME_ID, see the current parameters section in this wiki>
export EXTERNAL_IP=<YOUR_EXTERNAL_IP>
export ENTITY_DIR=/home/user/mainnet/entity
export DATA_DIR=/home/user/mainnet/node
export GENESIS_JSON=/home/user/mainnet/genesis.json
export RUNTIME_BINARY=/home/user/mainnet/oasis-ssvm-runtime
export SEED_NODE_ADDRESS=<OASIS_SEED_NODE_ADDRESS>
/home/user/mainnet/oasis-node \
--datadir $DATA_DIR \
--log.level warn \
--log.format json \
--log.file $DATA_DIR/node.log \
--worker.registration.entity $ENTITY_DIR/entity.json \
--genesis.file $GENESIS_JSON \
--worker.compute.enabled \
--worker.executor.schedule_check_tx.enabled \
--runtime.provisioner unconfined \
--runtime.supported $RUNTIME_ID \
--runtime.history.tag_indexer.backend bleve \
--runtime.paths $RUNTIME_ID=$RUNTIME_BINARY \
--debug.dont_blame_oasis \
--consensus.tendermint.p2p.seed $SEED_NODE_ADDRESS \
--consensus.tendermint.core.listen_address tcp://0.0.0.0:26656 \
--worker.client.addresses $EXTERNAL_IP:9100 \
--worker.p2p.addresses $EXTERNAL_IP:9200
- Since your private key is not on any online machine, you should sign this transaction offline.
- This step is to add your new ParaTime node to your entity.
- Join all your node IDs (Including your validator node ID and other node IDs) by
,
to get$NODE_IDS
.
Please notice that the node ID will be generated automatically after running the oasis-node command for the first time.
From $DATA_DIR/identity_pub.pem
. Or /home/user/mainnet/node/identity_pub.pem
in this guide.
cat $DATA_DIR/identity_pub.pem
(output below)
-----BEGIN ED25519 PUBLIC KEY-----
k3g1TUDwhI5ELAVxaomUq5zKEGnzTF74QgIdyTdYadM=
-----END ED25519 PUBLIC KEY-----
export NONCE=<YOUR_NONCE>
export ENTITY_DIR=/home/user/mainnet/entity/
export GENESIS_JSON=/home/user/mainnet/genesis.json
export OUTPUT_TX=/home/user/update_entity.tx
export NODE_IDS=<VALIDATOR_NODE_ID>,<PARATIME_NODE_ID>,<OTHER_NODE_ID_1>,...,<OTHER_NODE_ID_N>
oasis-node registry entity update \
--signer.dir $ENTITY_DIR \
--entity.node.id $NODE_IDS
oasis-node registry entity gen_register \
--genesis.file $GENESIS_JSON \
--signer.backend file \
--signer.dir $ENTITY_DIR \
--transaction.file $OUTPUT_TX \
--transaction.fee.gas 7000 \
--transaction.fee.amount 0 \
--transaction.nonce $NONCE
oasis-node consensus submit_tx \
--transaction.file /path/to/your/tx/file/in/previous/step \
--address unix:/home/user/mainnet/node/internal.sock
oasis-node control is-synced -a unix:/home/user/mainnet/node/internal.sock
(output below)
node completed initial syncing
If your paratime node is registered, you can find an external runtimes
information in the registration
like this:
oasis-node control status -address unix:/home/user/mainnet/node/internal.sock
(output below)
{
"software_version": "21.1.1",
"identity": {
...(omitted)...
},
"consensus": {
...(omitted)...
},
"runtimes": {
...(omitted)...
},
"registration": {
"last_registration": "2020-11-26T05:13:31Z",
"descriptor": {
...(omitted)...
},
"p2p": {
...(omitted)...
},
"consensus": {
...(omitted)...
},
"runtimes": [
{
"id": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/wQ=",
"version": {
"minor": 3
},
"capabilities": {},
"extra_info": null
}
],
"roles": 3
}
}
}
Or you can check if your paratime node is registered on http://status.oasiseth.org/.
When you start a node. The expected output will be the following:
{"msg":"Runtime is starting","level":"INFO","ts":"2021-03-04T22:30:37.631767781+00:00","module":"runtime"}
{"msg":"Establishing connection with the worker host","level":"INFO","ts":"2021-03-04T22:30:37.649013270+00:00","module":"runtime"}
{"msg":"Starting protocol handler","level":"INFO","ts":"2021-03-04T22:30:37.650019333+00:00","module":"runtime/protocol"}
{"msg":"Starting the runtime dispatcher","level":"INFO","ts":"2021-03-04T22:30:37.762550640+00:00","module":"runtime/dispatcher"}
...(No more output)...
If your paratime node is in the working committee, you will receive more output like this:
{"msg":"Received transaction batch request","level":"DEBG","ts":"2021-03-06T02:41:03.403869523+00:00","module":"runtime/dispatcher","check_only":true,"round":48,"state_root":"7c26ad41aeea0dea16422fcd8e5acc2fd2544c1d9e6e6bcca79dea98f3aadca3"}
{"msg":"Computing new block","level":"INFO","ts":"2021-03-06T02:41:03.406994419+00:00","module":"ethereum/block","round":48}
{"msg":"Commiting state into storage","level":"INFO","ts":"2021-03-06T02:41:03.705115962+00:00","module":"ethereum/block"}
{"msg":"Block finalized","level":"INFO","ts":"2021-03-06T02:41:03.706182128+00:00","module":"ethereum/block"}
{"msg":"Transaction batch check complete","level":"DEBG","ts":"2021-03-06T02:41:03.706497048+00:00","module":"runtime/dispatcher"}
{"msg":"Received transaction batch request","level":"DEBG","ts":"2021-03-06T02:41:08.569570167+00:00","module":"runtime/dispatcher","check_only":false,"round":49,"state_root":"7c26ad41aeea0dea16422fcd8e5acc2fd2544c1d9e6e6bcca79dea98f3aadca3"}
{"msg":"Computing new block","level":"INFO","ts":"2021-03-06T02:41:08.570749741+00:00","module":"ethereum/block","round":49}
{"msg":"Commiting state into storage","level":"INFO","ts":"2021-03-06T02:41:08.578648135+00:00","module":"ethereum/block"}
{"msg":"Block finalized","level":"INFO","ts":"2021-03-06T02:41:08.579089563+00:00","module":"ethereum/block"}
{"msg":"Transaction batch execution complete","level":"DEBG","ts":"2021-03-06T02:41:08.580159230+00:00","module":"runtime/dispatcher","state_root":"e51f19c24c7c3d0da88f1fbc38062bde96945dbf769567a7a0f6f86d95741c34","io_root":"f28478803a9906baeaa35ac811dc43b1415bd272f1761ab380bf0720335c8b93","previous_hash":"d2d203b2b6aed835c8515be7e613c0d948920ed03e0fa1e8faa8f26881bef011"}
You can retrieve a list of online OasisEth Paratime Nodes via the following command:
./oasis-node registry node list -v -a unix:/path/to/your/node/internal.sock | grep "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/wQ="
Then, you will get multiple OasisEth paratime nodes in this format:
{
"v":1,
"id":"FN4/IFFACdGqnp+hRDgaU0IVnRnILNU97SFyPXBEpk0=",
"entity_id":"UC0Aaa2Czc25G5+X2HOio6H0wQuBJR550chCE2kieN0=",
"expiration":3777,
"tls":{
"pub_key":"jHYVhcT8A/wCJDGi1I9y6VyQLER4D9Z0CZrub1g1DCk=",
"next_pub_key":"B4Z9WcS0h6b+dltzAbHVrkPuzJn2GM7AKr5kalw76lA=",
"addresses":[
"jHYVhcT8A/wCJDGi1I9y6VyQLER4D9Z0CZrub1g1DCk=@52.187.67.213:9100",
"B4Z9WcS0h6b+dltzAbHVrkPuzJn2GM7AKr5kalw76lA=@52.187.67.213:9100"
]
},
"p2p":{
"id":"71e8iRDjsTcvON1a9paAkYlst7enswVBjCGiT250IUc=",
"addresses":[
"52.187.67.213:9200"
]
},
"consensus":{
"id":"RFsavPSKFe5basHWfJBoWdnBigKVI76XbC41biT4Qhs=",
"addresses":null
},
"runtimes":[
{
"id":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/wQ=",
"version":{
"minor":3
},
"capabilities":{
},
"extra_info":null
}
],
"roles":3
}
If your node is register and online. You can find your node id
in one of the lines.
For example, one of our OasisEth paratime node ids is FN4/IFFACdGqnp+hRDgaU0IVnRnILNU97SFyPXBEpk0=
.
We provide a simple website to show the current network status: http://status.oasiseth.org/.
You can find the current whitelist and all registered paratime nodes.
Awesome! You already set up the paratime node!
We don't recommend you run the web3 RPC service within the paratime node, but you can still do that.
If you are going to provide the web3 RPC service for public usage, please refer to the Part 2 to set up a single web3 RPC service node for stability and scalability.
# run gateway
export RUNTIME_ID=<RUNTIME_ID>
gateway \
--interface 0.0.0.0 \ # Remove this line, if you don't want to expose your gateway.
--node-address unix:/path/to/your/internal.sock \
--runtime-id $RUNTIME_ID
Prepare the environment and download pre-built binaries
Our pre-built binaries only support Ubuntu 18.04 and Ubuntu 20.04
Please use the Ubuntu 18.04/20.04 to set up a machine.
The libssvm-evmc.so
MUST be put into /ssvm
folder.
So you will need to create a folder at path /ssvm
and download ssvm dependency into this folder.
mkdir /ssvm \
&& cd /ssvm \
&& curl -LO https://github.com/second-state/ssvm-evmc/releases/download/evmc6-0.1.1-rc1/libssvm-evmc.so
In the following guide, we use /home/user/mainnet
as an example. Feel free to replace this path with your own /path/to/mainnet/folder/
.
mkdir -p /home/user/mainnet
cd /home/user/mainnet
# Oasis-Core
wget <Oasis-Core-Binary URL, see the network parameter on the above section>
# Mainnet genesis
wget <Oasis-Genesis URL, see the network parameter on the above section>
# oasis-ssvm-runtime
wget <Oasis-SSVM-runtime URL, see the network parameter on the above section>
tar zxvf oasis-ssvm-runtime.tgz
rm oasis-ssvm-runtime.tgz
/ssvm/libssvm-evmc.so
/home/user/mainnet
├── genesis.json
├── gateway
├── oasis-node
└── oasis-ssvm-runtime
- Make sure you should use a proper IP for
$EXTERNAL_IP
. - You should expose the following ports:
9100
,9200
,26656
,8545
-
9100
: This port is for the Oasis ParaTime gateway connection. -
9200
: This port is for the Oasis ParaTime P2P connection. -
26656
: This port is for tendermint consensus gossip. -
8545
: This port is for the JSONRPC endpoint.
-
mkdir -m 700 -p /home/user/mainnet/node
datadir: /home/user/mainnet/node
log:
level: warn
format: json
file: /home/user/mainnet/client.log
genesis:
file: /home/user/mainnet/genesis.json
storage:
backend: client
runtime:
supported: "<RUNTIME_ID>"
history:
tag_indexer:
backend: bleve
worker:
p2p:
enabled: true
consensus:
tendermint:
core:
listen_address: tcp://0.0.0.0:26656
p2p:
seed:
- "<OASIS_SEED_NODE_ADDRESS>"
This is only running the oasis-node
cd /home/user/mainnet
# run node
oasis-node --config ./config.lightweight.yml &
# run gateway
export RUNTIME_ID=<RUNTIME_ID>
gateway \
--interface 0.0.0.0 \ # Remove this line, if you don't want to expose your gateway.
--node-address unix:./node/internal.sock \
--runtime-id $RUNTIME_ID
- Stop your paratime node
- Download the latest binary releases of oasis-node, ssvm-evmc, oasis-ssvm-runtime, and gateway
- Replace the old ones with the new ones which are downloaded in step 2
- Start your node
- Stop your gateway
- Stop your web3 RPC node
- Download the latest binary releases of oasis-node, ssvm-evmc, oasis-ssvm-runtime, and gateway
- Replace the old ones with the new ones which are downloaded in step 3
- Start your web3 RPC node
- Start your gateway
Please refer to the Oasis: Create your entity guide to generate your own entity.
The entity id is required. We must add your entity id to our ParaTime allowlist to make sure you have permission to run a runtime node.
You need an Oasis node to submit register node
transactions. If you don't have one of these nodes, please follow the oasis docs to set up a Validator Node or a Non-Validator Node
If you get one of the following error messages:
{"caller":"registry.go:435","err":"consensus: no committed blocks","level":"error","module":"registry/tendermint","msg":"runtime notifier: unable to get a list of runtimes","ts":"2021-01-11T16:26:31.847903663Z"}
{"caller":"registry.go:423","err":"consensus: no committed blocks","level":"error","module":"registry/tendermint","msg":"node list notifier: unable to get a list of nodes","ts":"2021-01-11T16:26:31.885906513Z"}
{"caller":"scheduler.go:152","err":"consensus: no committed blocks","level":"error","module":"scheduler/tendermint","msg":"couldn't get current committees. won't send them. good luck to the subscriber","ts":"2021-01-11T16:26:31.902684412Z"}
{"caller":"grpc.go:202","err":"context canceled","level":"error","method":"/oasis-core.NodeController/WaitSync","module":"grpc/internal","msg":"request failed","req_seq":2,"ts":"2021-01-11T17:44:36.151422213Z"}
Please check your seed is setting to the right one.
oasis-node ...
--consensus.tendermint.p2p.seed $SEED_NODE_ADDRESS
consensus:
tendermint:
...
p2p:
seed:
- "<SEED_NODE_ADDRESS>"
When you receive this error message, that means your node data is somewhere broken. You should remove your DATA_DIR
and re-sync your node again.
When you check the oasis-node control status
, you can find your registration information and the worker status. You may find that your registration is done like this:
(...omitted...)
"registration": {
"last_registration": "2021-02-08T10:38:06+01:00",
"descriptor": {
"v": 1,
"id": "...",
"entity_id": "...",
"expiration": 3148,
(...omitted...)
"runtimes": [
{
"id": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/wQ=",
"version": {
"minor": 3
},
"capabilities": {},
"extra_info": null
}
],
"roles": 3
}
}
(...omitted...)
But you still find that:
"is_validator": false
"executor_role": "invalid",
"storage_role": "invalid",
"is_txn_scheduler": false,
- The
is_validator
means the current node is oasis validator or not. This should be always false because the paratime node is not a validator node. - The
executor_role
means the current node is in the executor committee in this epoch period. If it's in the executor committee, the value will beworker
, otherwise, the value will beinvalid
. - The
storage_role
means the current node is in the storage committee in this epoch period. If it's in the storage committee, the value will beworker
, otherwise, the value will beinvalid
. - The
is_txn_scheduler
means the current node is in the executor/storage committee, and this node is the current block proposer.
Please notice that even your entity is in the paratime whitelist, you may not in each executor
committee or storage
committee. The committee will be chosen from the whitelist every epoch time.