-
Notifications
You must be signed in to change notification settings - Fork 615
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Running relayer: documentation (#2829)
* running relayer documentation * fix review comments * also update tag * typo * Apply suggestions from code review --------- Co-authored-by: Adrian Catangiu <adrian@parity.io>
- Loading branch information
Showing
4 changed files
with
344 additions
and
0 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,344 @@ | ||
# Running your own bridge relayer | ||
|
||
:warning: :construction: Please read the [Disclaimer](#disclaimer) section first :construction: :warning: | ||
|
||
## Disclaimer | ||
|
||
There are several things you should know before running your own relayer: | ||
|
||
- initial bridge version (we call it bridges v1) supports any number of relayers, but **there's no guaranteed | ||
compensation** for running a relayer and/or submitting valid bridge transactions. Most probably you'll end up | ||
spending more funds than getting from rewards - please accept this fact; | ||
|
||
- even if your relayer has managed to submit a valid bridge transaction that has been included into the bridge | ||
hub block, there's no guarantee that you will be able to claim your compensation for that transaction. That's | ||
because compensations are paid from the account, controlled by relay chain governance and it could have no funds | ||
to compensate your useful actions. We'll be working on a proper process to resupply it on-time, but we can't | ||
provide any guarantee until that process is well established. | ||
|
||
## A Brief Introduction into Relayers and our Compensations Scheme | ||
|
||
Omitting details, relayer is an offchain process that is connected to both bridged chains. It looks at the | ||
outbound bridge messages queue and submits message delivery transactions to the target chain. There's a lot | ||
of details behind that simple phrase - you could find more info in the | ||
[High-Level Bridge Overview](./high-level-overview.md) document. | ||
|
||
Reward that is paid to relayer has two parts. The first part static and is controlled by the governance. | ||
It is rather small initially - e.g. you need to deliver `10_000` Kusama -> Polkadot messages to gain single | ||
KSM token. | ||
|
||
The other reward part is dynamic. So to deliver an XCM message from one BridgeHub to another, we'll need to | ||
submit two transactions on different chains. Every transaction has its cost, which is: | ||
|
||
- dynamic, because e.g. message size can change and/or fee factor of the target chain may change; | ||
|
||
- quite large, because those transactions are quite heavy (mostly in terms of size, not weight). | ||
|
||
We are compensating the cost of **valid**, **minimal** and **useful** bridge-related transactions to | ||
relayer, that has submitted such transaction. Valid here means that the transaction doesn't fail. Minimal | ||
means that all data within transaction call is actually required for the transaction to succeed. Useful | ||
means that all supplied data in transaction is new and yet unknown to the target chain. | ||
|
||
We have implemented a relayer that is able to craft such transactions. The rest of document contains a detailed | ||
information on how to deploy this software on your own node. | ||
|
||
## Relayers Concurrency | ||
|
||
As it has been said above, we are not compensating cost of transactions that are not **useful**. For | ||
example, if message `100` has already been delivered from Kusama Bridge Hub to Polkadot Bridge Hub, then another | ||
transaction that delivers the same message `100` won't be **useful**. Hence, no compensation to relayer that | ||
has submitted that second transaction. | ||
|
||
But what if there are several relayers running? They are noticing the same queued message `100` and | ||
simultaneously submit identical message delivery transactions. You may expect that there'll be one lucky | ||
relayer, whose transaction would win the "race" and which will receive the compensation and reward. And | ||
there'll be several other relayers, losing some funds on their unuseful transactions. | ||
|
||
But actually, we have a solution that invalidates transactions of "unlucky" relayers before they are | ||
included into the block. So at least you may be sure that you won't waste your funds on duplicate transactions. | ||
|
||
<details> | ||
<summary>Some details?</summary> | ||
|
||
All **unuseful** transactions are rejected by our | ||
[transaction extension](https://github.com/paritytech/polkadot-sdk/blob/master/bridges/bin/runtime-common/src/refund_relayer_extension.rs), | ||
which also handles transaction fee compensations. You may find more info on unuseful (aka obsolete) transactions | ||
by lurking in the code. | ||
|
||
We also have the WiP prototype of relayers coordination protocol, where relayers will get some guarantee | ||
that their transactions will be prioritized over other relayers transactions at their assigned slots. | ||
That is planned for the future version of bridge and the progress is | ||
[tracked here](https://github.com/paritytech/parity-bridges-common/issues/2486). | ||
|
||
</details> | ||
|
||
## Prerequisites | ||
|
||
Let's focus on the bridge between Polkadot and Kusama Bridge Hubs. Let's also assume that we want to start | ||
a relayer that "serves" an initial lane [`0x00000001`](https://github.com/polkadot-fellows/runtimes/blob/9ce1bbbbcd7843b3c76ba4d43c036bc311959e9f/system-parachains/bridge-hubs/bridge-hub-kusama/src/bridge_to_polkadot_config.rs#L54). | ||
|
||
<details> | ||
<summary>Lane?</summary> | ||
|
||
Think of lane as a queue of messages that need to be delivered to the other/bridged chain. The lane is | ||
bidirectional, meaning that there are four "endpoints". Two "outbound" endpoints (one at every chain), contain | ||
messages that need to be delivered to the bridged chain. Two "inbound" are accepting messages from the bridged | ||
chain and also remember the relayer, who has delivered message(s) to reward it later. | ||
|
||
</details> | ||
|
||
The same steps may be performed for other lanes and bridges as well - you'll just need to change several parameters. | ||
|
||
So to start your relayer instance, you'll need to prepare: | ||
|
||
- an address of ws/wss RPC endpoint of the Kusama relay chain; | ||
|
||
- an address of ws/wss RPC endpoint of the Polkadot relay chain; | ||
|
||
- an address of ws/wss RPC endpoint of the Kusama Bridge Hub chain; | ||
|
||
- an address of ws/wss RPC endpoint of the Polkadot Bridge Hub chain; | ||
|
||
- an account on Kusama Bridge Hub; | ||
|
||
- an account on Polkadot Bridge Hub. | ||
|
||
For RPC endpoints, you could start your own nodes, or use some public community nodes. Nodes are not meant to be | ||
archive or provide access to insecure RPC calls. | ||
|
||
To create an account on Bridge Hubs, you could use XCM teleport functionality. E.g. if you have an account on | ||
the relay chain, you could use the `teleportAssets` call of `xcmPallet` and send asset | ||
`V3 { id: Concrete(0, Here), Fungible: <your-amount> }` to beneficiary `V3(0, X1(AccountId32(<your-account>)))` | ||
on destination `V3(0, X1(Parachain(1002)))`. To estimate amounts you need, please refer to the [Costs](#costs) | ||
section of the document. | ||
|
||
## Registering your Relayer Account (Optional, But Please Read) | ||
|
||
Bridge transactions are quite heavy and expensive. We want to minimize block space that can be occupied by | ||
invalid bridge transactions and prioritize valid transactions over invalid. That is achieved by **optional** | ||
relayer registration. Transactions, signed by relayers with active registration, gain huge priority boost. | ||
In exchange, such relayers may be slashed if they submit **invalid** or **non-minimal** transaction. | ||
|
||
Transactions, signed by relayers **without** active registration, on the other hand, receive no priority | ||
boost. It means that if there is active registered relayer, most likely all transactions from unregistered | ||
will be counted as **unuseful**, not included into the block and unregistered relayer won't get any reward | ||
for his operations. | ||
|
||
Before registering, you should know several things about your funds: | ||
|
||
- to register, you need to hold significant amount of funds on your relayer account. As of now, it is | ||
[100 KSM](https://github.com/polkadot-fellows/runtimes/blob/9ce1bbbbcd7843b3c76ba4d43c036bc311959e9f/system-parachains/bridge-hubs/bridge-hub-kusama/src/bridge_to_polkadot_config.rs#L71C14-L71C43) | ||
for registration on Kusama Bridge Hub and | ||
[500 DOT](https://github.com/polkadot-fellows/runtimes/blob/9ce1bbbbcd7843b3c76ba4d43c036bc311959e9f/system-parachains/bridge-hubs/bridge-hub-polkadot/src/bridge_to_kusama_config.rs#L71C14-L71C43) | ||
for registration on Polkadot Bridge Hub; | ||
|
||
- when you are registered, those funds are reserved on relayer account and you can't transfer them. | ||
|
||
The registration itself, has three states: active, inactive or expired. Initially, it is active, meaning that all | ||
your transactions that are **validated** on top of block, where it is active get priority boost. Registration | ||
becomes expired when the block with the number you have specified during registration is "mined". It is the | ||
`validTill` parameter of the `register` call (see below). After that `validTill` block, you may unregister and get | ||
your reserved funds back. There's also an intermediate point between those blocks - it is the `validTill - LEASE`, | ||
where `LEASE` is the the chain constant, controlled by the governance. Initially it is set to `300` blocks. | ||
All your transactions, **validated** between the `validTill - LEASE` and `validTill` blocks do not get the | ||
priority boost. Also, it is forbidden to specify `validTill` such that the `validTill - currentBlock` is less | ||
than the `LEASE`. | ||
|
||
<details> | ||
<summary>Example?</summary> | ||
|
||
| Bridge Hub Block | Registration State | Comment | | ||
| ----------------- | ------------------ | ------------------------------------------------------ | | ||
| 100 | Active | You have submitted a tx with the `register(1000)` call | | ||
| 101 | Active | Your message delivery transactions are boosted | | ||
| 102 | Active | Your message delivery transactions are boosted | | ||
| ... | Active | Your message delivery transactions are boosted | | ||
| 700 | Inactive | Your message delivery transactions are not boosted | | ||
| 701 | Inactive | Your message delivery transactions are not boosted | | ||
| ... | Inactive | Your message delivery transactions are not boosted | | ||
| 1000 | Expired | Your may submit a tx with the `deregister` call | | ||
|
||
</details> | ||
|
||
So once you have enough funds on your account and have selected the `validTill` parameter value, you | ||
could use the Polkadot JS apps to submit an extrinsic. If you want priority boost for your transactions | ||
on the Kusama Bridge Hub, open the | ||
[Polkadot JS Apps](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-bridge-hub-rpc.polkadot.io#/extrinsics) | ||
and submit the `register` extrinsic from the `bridgeRelayers` pallet: | ||
|
||
![Register Extrinsic](./bridge-relayers-register.png) | ||
|
||
To deregister, submit the simple `deregister` extrinsic when registration is expired: | ||
|
||
![Deregister Extrinsic](./bridge-relayers-deregister.png) | ||
|
||
At any time, you can prolong your registration by calling the `register` with the larger `validTill`. | ||
|
||
## Costs | ||
|
||
Your relayer account (on both Bridge Hubs) must hold enough funds to be able to pay costs of bridge | ||
transactions. If your relayer behaves correctly, those costs will be compensated and you will be | ||
able to claim it later. | ||
|
||
**IMPORTANT**: you may add tip to your bridge transactions to boost their priority. But our | ||
compensation mechanism never refunds transaction tip, so all tip tokens will be lost. | ||
|
||
<details> | ||
<summary>Types of bridge transactions</summary> | ||
|
||
There are two types of bridge transactions: | ||
|
||
- message delivery transaction brings queued message(s) from one Bridge Hub to another. We record | ||
the fact that this specific (your) relayer has delivered those messages; | ||
|
||
- message confirmation transaction confirms that some message have been delivered and also brings | ||
back information on how many messages (your) relayer has delivered. We use this information later | ||
to register delivery rewards on the source chain. | ||
|
||
Several messages/confirmations may be included in a single bridge transaction. Apart from this | ||
data, bridge transaction may include finality and storage proofs, required to prove authenticity of | ||
this data. | ||
|
||
</details> | ||
|
||
To deliver and get reward for a single message, the relayer needs to submit two transactions. One | ||
at the source Bridge Hub and one at the target Bridge Hub. Below are costs for Polkadot <> Kusama | ||
messages (as of today): | ||
|
||
- to deliver a single Polkadot -> Kusama message, you would need to pay around `0.06 KSM` at Kusama | ||
Bridge Hub and around `1.62 DOT` at Polkadot Bridge Hub; | ||
|
||
- to deliver a single Kusama -> Polkadot message, you would need to pay around `1.70 DOT` at Polkadot | ||
Bridge Hub and around `0.05 KSM` at Kusama Bridge Hub. | ||
|
||
Those values are not constants - they depend on call weights (that may change from release to release), | ||
on transaction sizes (that depends on message size and chain state) and congestion factor. In any | ||
case - it is your duty to make sure that the relayer has enough funds to pay transaction fees. | ||
|
||
## Claiming your Compensations and Rewards | ||
|
||
Hopefully you have successfully delivered some messages and now can claim your compensation and reward. | ||
This requires submitting several transactions. But first, let's check that you actually have something to | ||
claim. For that, let's check the state of the pallet that tracks all rewards. | ||
|
||
To check your rewards at the Kusama Bridge Hub, go to the | ||
[Polkadot JS Apps](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-bridge-hub-rpc.polkadot.io#/chainstate) | ||
targeting Kusama Bridge Hub, select the `bridgeRelayers` pallet, choose `relayerRewards` map and | ||
your relayer account. Then: | ||
|
||
- set the `laneId` to `0x00000001` | ||
|
||
- set the `bridgedChainId` to `bhpd`; | ||
|
||
- check the both variants of the `owner` field: `ThisChain` is used to pay for message delivery transactions | ||
and `BridgedChain` is used to pay for message confirmation transactions. | ||
|
||
If check shows that you have some rewards, you can craft the claim transaction, with similar parameters. | ||
For that, go to `Extrinsics` tab of the | ||
[Polkadot JS Apps](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-bridge-hub-rpc.polkadot.io#/extrinsics) | ||
and submit the following transaction (make sure to change `owner` before): | ||
|
||
![Claim Rewards Extrinsic](./bridge-relayers-claim-rewards.png) | ||
|
||
To claim rewards on Polkadot Bridge Hub you can follow the same process. The only difference is that you | ||
need to set value of the `bridgedChainId` to `bhks`. | ||
|
||
## Starting your Relayer | ||
|
||
### Starting your Rococo <> Westend Relayer | ||
|
||
We are using the following docker image to run our own Rococo <> Westend relayer: | ||
|
||
[paritytech/bridges-common-relay:v2024-02-16-rococo-westend-relayer](https://hub.docker.com/layers/paritytech/bridges-common-relay/v2024-02-16-rococo-westend-relayer/images/sha256-5ba4260d6bc2ac9e97c2e05d875f6dc44ee9265011311f268b66b8a5a672ada3?context=explore) | ||
|
||
You can grab the sources and build it manually: | ||
|
||
[v2024-02-16-rococo-westend-relayer](https://github.com/paritytech/parity-bridges-common/releases/tag/v2024-02-16-rococo-westend-relayer) | ||
|
||
Once you have the docker, update variables and run the following script: | ||
```sh | ||
export DOCKER_IMAGE=<image-of-substrate-relay> | ||
|
||
export ROCOCO_HOST=<rococo-ws-rpc-host-here> | ||
export ROCOCO_PORT=<rococo-ws-rpc-port-here> | ||
# or set it to '--rococo-secure' if wss is used above | ||
export ROCOCO_IS_SECURE= | ||
export BRIDGE_HUB_ROCOCO_HOST=<bridge-hub-rococo-ws-rpc-host-here> | ||
export BRIDGE_HUB_ROCOCO_PORT=<bridge-hub-rococo-ws-rpc-port-here> | ||
# or set it to '--bridge-hub-rococo-secure' if wss is used above | ||
export BRIDGE_HUB_ROCOCO_IS_SECURE= | ||
export BRIDGE_HUB_ROCOCO_KEY_FILE=<absolute-path-to-file-with-account-key-at-bridge-hub-rococo> | ||
|
||
export WESTEND_HOST=<westend-wss-rpc-host-here> | ||
export WESTEND_PORT=<westend-wss-rpc-port-here> | ||
# or set it to '--westend-secure' if wss is used above | ||
export WESTEND_IS_SECURE= | ||
export BRIDGE_HUB_WESTEND_HOST=<bridge-hub-westend-ws-rpc-host-here> | ||
export BRIDGE_HUB_WESTEND_PORT=<bridge-hub-westend-ws-rpc-port-here> | ||
# or set it to '--bridge-hub-westend-secure ' if wss is used above | ||
export BRIDGE_HUB_WESTEND_IS_SECURE= | ||
export BRIDGE_HUB_WESTEND_KEY_FILE=<absolute-path-to-file-with-account-key-at-bridge-hub-westend> | ||
|
||
# you can get extended relay logs (e.g. for debugging issues) by passing `-e RUST_LOG=bridge=trace` | ||
# argument to the `docker` binary | ||
docker run \ | ||
-v $BRIDGE_HUB_ROCOCO_KEY_FILE:/bhr.key \ | ||
-v $BRIDGE_HUB_WESTEND_KEY_FILE:/bhw.key \ | ||
$DOCKER_IMAGE \ | ||
relay-headers-and-messages bridge-hub-rococo-bridge-hub-westend \ | ||
--rococo-host $ROCOCO_HOST \ | ||
--rococo-port $ROCOCO_PORT \ | ||
$ROCOCO_IS_SECURE \ | ||
--rococo-version-mode Auto \ | ||
--bridge-hub-rococo-host $BRIDGE_HUB_ROCOCO_HOST \ | ||
--bridge-hub-rococo-port $BRIDGE_HUB_ROCOCO_PORT \ | ||
$BRIDGE_HUB_ROCOCO_IS_SECURE \ | ||
--bridge-hub-rococo-version-mode Auto \ | ||
--bridge-hub-rococo-signer-file /bhr.key \ | ||
--bridge-hub-rococo-transactions-mortality 16 \ | ||
--westend-host $WESTEND_HOST \ | ||
--westend-port $WESTEND_PORT \ | ||
$WESTEND_IS_SECURE \ | ||
--westend-version-mode Auto \ | ||
--bridge-hub-westend-host $BRIDGE_HUB_WESTEND_HOST \ | ||
--bridge-hub-westend-port $BRIDGE_HUB_WESTEND_PORT \ | ||
$BRIDGE_HUB_WESTEND_IS_SECURE \ | ||
--bridge-hub-westend-version-mode Auto \ | ||
--bridge-hub-westend-signer-file /bhw.key \ | ||
--bridge-hub-westend-transactions-mortality 16 \ | ||
--lane 00000002 | ||
``` | ||
|
||
### Starting your Polkadot <> Kusama Relayer | ||
|
||
*Work in progress, coming soon* | ||
|
||
### Watching your relayer state | ||
|
||
Our relayer provides some Prometheus metrics that you may convert into some fancy Grafana dashboards | ||
and alerts. By default, metrics are exposed at port `9616`. To expose endpoint to the localhost, change | ||
the docker command by adding following two lines: | ||
|
||
```sh | ||
docker run \ | ||
.. | ||
-p 127.0.0.1:9616:9616 \ # tell Docker to bind container port 9616 to host port 9616 | ||
# and listen for connections on the host' localhost interface | ||
.. | ||
$DOCKER_IMAGE \ | ||
relay-headers-and-messages bridge-hub-rococo-bridge-hub-westend \ | ||
--prometheus-host 0.0.0.0 \ # tell `substrate-relay` binary to accept Prometheus endpoint | ||
# connections from everywhere | ||
.. | ||
``` | ||
|
||
You can find more info on configuring Prometheus and Grafana in the | ||
[Monitor your node](https://wiki.polkadot.network/docs/maintain-guides-how-to-monitor-your-node) | ||
guide from Polkadot wiki. | ||
|
||
We have our own set of Grafana dashboards and alerts. You may use them for inspiration. | ||
Please find them in this folder: | ||
|
||
- for Rococo <> Westend bridge: [rococo-westend](https://github.com/paritytech/parity-bridges-common/tree/master/deployments/bridges/rococo-westend). | ||
|
||
- for Polkadot <> Kusama bridge: *work in progress, coming soon* |