Update project level docs

README.md

@@ -10,6 +10,10 @@ Substrate chains.
 
 🚧 The bridges are currently under construction - a hardhat is recommended beyond this point 🚧
 
+## Past, modern and future of the project
+
+The project in its current state provides a cross-consensus transport for XCM messages. It 
+
 **IMPORTANT**: this documentation is outdated and it is mostly related to the previous version of our
 bridge. Right there's an ongoing work to make our bridge work with XCM messages. Old bridge is still
 available at [encoded-calls-messaging](https://github.com/paritytech/parity-bridges-common/releases/tag/encoded-calls-messaging)
@@ -97,7 +101,7 @@ description of the bridge interaction.
 
 ## Project Layout
 
-Here's an overview of how the project is laid out. The main bits are the `node`, which is the actual
+Here's an overview of how the project is laid out. The main bits are the `bin`, which is the actual
 "blockchain", the `modules` which are used to build the blockchain's logic (a.k.a the runtime) and
 the `relays` which are used to pass messages between chains.
 
@@ -106,16 +110,16 @@ the `relays` which are used to pass messages between chains.
 │  └── ...
 ├── deployments     // Useful tools for deploying test networks
 │  └──  ...
-├── diagrams        // Pretty pictures of the project architecture
-│  └──  ...
 ├── modules         // Substrate Runtime Modules (a.k.a Pallets)
+│  ├── beefy        // On-Chain BEEFY Light Client (in progress)
 │  ├── grandpa      // On-Chain GRANDPA Light Client
 │  ├── messages     // Cross Chain Message Passing
-│  ├── dispatch     // Target Chain Message Execution
+│  ├── parachains   // On-Chain Parachains Light Client
+│  ├── relayers     // Relayer rewards registry
 │  └──  ...
 ├── primitives      // Code shared between modules, runtimes, and relays
 │  └──  ...
-├── relays          // Application for sending headers and messages between chains
+├── relays          // Application for sending finality proofs and messages between chains
 │  └──  ...
 └── scripts         // Useful development and maintenance scripts
 ```
@@ -127,8 +131,11 @@ on each side of the bridge (source and target chain).
 
 There are 2 ways to run the bridge, described below:
 
-- building & running from source
-- running a Docker Compose setup (recommended).
+- building & running from source: with this option, you'll be able to run the bridge between two standalone
+chains that are running GRANDPA finality gadget to achieve finality;
+
+- running a Docker Compose setup: this is a recommended option, where you'll see bridges with parachains,
+complex relays and more.
 
 ### Using the Source
 
@@ -204,7 +211,33 @@ You will also see the message lane relayers listening for new messages.
 
 To send a message see the ["How to send a message" section](#how-to-send-a-message).
 
-### Full Network Docker Compose Setup
+### How to send a message
+
+In this section we'll show you how to quickly send a bridge message. The message is just an encoded XCM
+`Trap(43)` message.
+
+```bash
+# In `parity-bridges-common` folder
+./scripts/send-message-from-millau-rialto.sh
+```
+
+After sending a message you will see the following logs showing a message was successfully sent:
+
+```
+INFO bridge Sending message to Rialto. Size: 5.
+TRACE bridge Sent transaction to Millau node: 0x5e68...
+```
+
+And at the Rialto node logs you'll something like this:
+
+```
+... runtime::bridge-dispatch: Going to execute message ([0, 0, 0, 0], 1) (...), Trap(43)])    
+... runtime::bridge-dispatch: Incoming message ([0, 0, 0, 0], 1) dispatched with result: Incomplete(2000000000, Trap(43))    
+``` 
+
+It means that the message has been delivered and successfully dispatched.
+
+## Full Network Docker Compose Setup
 
 For a more sophisticated deployment which includes bidirectional header sync, message passing,
 monitoring dashboards, etc. see the [Deployments README](./deployments/README.md).
@@ -220,24 +253,6 @@ docker run -p 30333:30333 -p 9933:9933 -p 9944:9944 \
   --rpc-cors=all --unsafe-rpc-external --unsafe-ws-external
 ```
 
-### How to send a message
-
-In this section we'll show you how to quickly send a bridge message, if you want to
-interact with and test the bridge see more details in [send message](./docs/send-message.md)
-
-```bash
-# In `parity-bridges-common` folder
-./scripts/send-message-from-millau-rialto.sh remark
-```
-
-After sending a message you will see the following logs showing a message was successfully sent:
-
-```
-INFO bridge Sending message to Rialto. Size: 286. Dispatch weight: 1038000. Fee: 275,002,568
-INFO bridge Signed Millau Call: 0x7904...
-TRACE bridge Sent transaction to Millau node: 0x5e68...
-```
-
 ## Community
 
 Main hangout for the community is [Element](https://element.io/) (formerly Riot). Element is a chat

deployments/README.md

@@ -1,11 +1,13 @@
 # Bridge Deployments
 
 ## Requirements
+
 Make sure to install `docker` and `docker-compose` to be able to run and test bridge deployments. If
 for whatever reason you can't or don't want to use Docker, you can find some scripts for running the
-bridge [here](https://github.com/svyatonik/parity-bridges-common.test).
+bridge [here](./local-scripts/).
 
 ## Networks
+
 One of the building blocks we use for our deployments are _networks_. A network is a collection of
 homogenous blockchain nodes. We have Docker Compose files for each network that we want to bridge.
 Each of the compose files found in the `./networks` folder is able to independently spin up a
@@ -18,6 +20,7 @@ docker-compose -f ./networks/rialto.yml up
 After running this command we would have a network of several nodes producing blocks.
 
 ## Bridges
+
 A _bridge_ is a way for several _networks_ to connect to one another. Bridge deployments have their
 own Docker Compose files which can be found in the `./bridges` folder. These Compose files typically
 contain bridge relayers, which are services external to blockchain nodes, and other components such
@@ -43,10 +46,11 @@ and Grafana. We cover these in more details in the [Monitoring](#monitoring) sec
 the monitoring Compose file is _not_ optional, and must be included for bridge deployments.
 
 ### Running and Updating Deployments
+
 We currently support three bridge deployments
 1. Rialto Substrate to Millau Substrate
 2. Rialto Parachain Substrate to Millau Substrate
-2. Westend Substrate to Millau Substrate
+2. Westend Substrate to Millau Substrate (only finality bridge)
 
 These bridges can be deployed using our [`./run.sh`](./run.sh) script.
 
@@ -71,11 +75,12 @@ You can also bring down a deployment using the script with the `stop` argument.
 ```
 
 ### Adding Deployments
+
 We need two main things when adding a new deployment. First, the new network which we want to
 bridge. A compose file for the network should be added in the `/networks/` folder. Secondly we'll
 need a new bridge Compose file in `./bridges/`. This should configure the bridge relayer nodes
 correctly for the two networks, and add any additional components needed for the deployment. If you
-want you can also add support in the `./run` script for the new deployment. While recommended it's
+want you can also add support in the `./run.sh` script for the new deployment. While recommended it's
 not strictly required.
 
 ## General Notes
@@ -111,8 +116,8 @@ is not recommended, because this may lead to nonces conflict.
 
 Following accounts are used when `rialto-millau` bridge is running:
 
-- Millau's `Rialto.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on Millau chain;
-- Rialto's `Millau.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on Rialto chain;
+- Millau's `Rialto.HeadersAndMessagesRelay1` signs complex headers+messages relay transactions on Millau chain;
+- Rialto's `Millau.HeadersAndMessagesRelay1` signs complex headers+messages relay transactions on Rialto chain;
 - Millau's `Rialto.MessagesSender` signs Millau transactions which contain messages for Rialto;
 - Rialto's `Millau.MessagesSender` signs Rialto transactions which contain messages for Millau;
 - Millau's `Rialto.OutboundMessagesRelay.Lane00000001` signs relay transactions with message delivery confirmations (lane 00000001) from Rialto to Millau;
@@ -136,18 +141,22 @@ Following accounts are used when `rialto-parachain-millau` bridge is running:
 - RialtoParachain's `Millau.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on RialtoParachain chain.
 
 ### Docker Usage
+
 When the network is running you can query logs from individual nodes using:
 
 ```bash
 docker logs rialto_millau-node-charlie_1 -f
 ```
 
+You may use the [dump-logs.sh](../scripts/dump-logs.sh) to dump logs of most of running containers.
+
 To kill all leftover containers and start the network from scratch next time:
 ```bash
 docker ps -a --format "{{.ID}}" | xargs docker rm # This removes all containers!
 ```
 
 ### Docker Compose Usage
+
 If you're not familiar with how to use `docker-compose` here are some useful commands you'll need
 when interacting with the bridge deployments:
 
@@ -172,10 +181,12 @@ docker-compose -f docker-compose.yml -f docker-compose.override.yml config > doc
 ```
 
 ## Docker and Git Deployment
+
 It is also possible to avoid using images from the Docker Hub and instead build
 containers from Git. There are two ways to build the images this way.
 
 ### Git Repo
+
 If you have cloned the bridges repo you can build local Docker images by running the following
 command at the top level of the repo:
 
@@ -189,16 +200,17 @@ This will build a local image of a particular component with a tag of
 You can configure the build using Docker
 [build arguments](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg).
 Here are the arguments currently supported:
-  - `BRIDGE_REPO`: Git repository of the bridge node and relay code
-  - `BRIDGE_HASH`: Commit hash within that repo (can also be a branch or tag)
-  - `ETHEREUM_REPO`: Git repository of the OpenEthereum client
-  - `ETHEREUM_HASH`: Commit hash within that repo (can also be a branch or tag)
   - `PROJECT`: Project to build withing bridges repo. Can be one of:
     - `rialto-bridge-node`
     - `millau-bridge-node`
+    - `rialto-parachain-collator`
     - `substrate-relay`
 
+You may use the [build-containers.sh](../scripts/build-containers.sh) script to build all available
+containers.
+
 ### GitHub Actions
+
 We have a nightly job which runs and publishes Docker images for the different nodes and relayers to
 the [ParityTech Docker Hub](https://hub.docker.com/u/paritytech) organization. These images are used
 for our ephemeral (temporary) test networks. Additionally, any time a tag in the form of `v*` is
@@ -209,6 +221,7 @@ With images built using either method, all you have to do to use them in a deplo
 `image` field in the existing Docker Compose files to point to the tag of the image you want to use.
 
 ### Monitoring
+
 [Prometheus](https://prometheus.io/) is used by the bridge relay to monitor information such as system
 resource use, and block data (e.g the best blocks it knows about). In order to visualize this data
 a [Grafana](https://grafana.com/) dashboard can be used.
@@ -223,6 +236,7 @@ dashboard can be accessed at `http://localhost:9090`. The Grafana dashboard can
 `http://localhost:3000`. Note that the default log-in credentials for Grafana are `admin:admin`.
 
 ### Environment Variables
+
 Here is an example `.env` file which is used for production deployments and network updates. For
 security reasons it is not kept as part of version control. When deploying a network this
 file should be correctly populated and kept in the appropriate [`bridges`](`./bridges`) deployment
@@ -247,13 +261,8 @@ UI_EXPECTED_ETHEREUM_NETWORK_ID=105
 
 ### UI
 
-Use [wss://rialto.bridges.test-installations.parity.io/](https://polkadot.js.org/apps/)
-as a custom endpoint for [https://polkadot.js.org/apps/](https://polkadot.js.org/apps/).
-
-### Polkadot.js UI
-
-To teach the UI decode our custom types used in the pallet, go to: `Settings -> Developer`
-and import the [`./types.json`](./types.json)
+Use [wss://wss.rialto.brucke.link](https://polkadot.js.org/apps/) as a custom endpoint for
+[https://polkadot.js.org/apps/](https://polkadot.js.org/apps/).
 
 ## Scripts
 

docs/complex-relay.html

@@ -0,0 +1,85 @@
+<!DOCTYPE html>
+<html>
+	<head>
+		<meta charset="utf-8">
+		<meta name="viewport" content="width=device-width">
+		<title>Complex Relay</title>
+	</head>
+	<body>
+		<h1>Complex Relay</h1>
+		<p>
+			Both Source Chain and Target Chains have Bridge Messages pallets deployed. They also have required
+			finality pallets deployed - we don't care about finality type here - they can be either Bridge GRANDPA,
+			or Bridge Parachains finality pallets, or any combination of those.<br/>
+		</p>
+		<p>
+			There are 4-6 relayer subprocesses inside the Complex Relayer. They include two message relayers,
+			serving the lane in both directions and 2-4 Complex Relayers (depending on the finality type of Source
+			and Target Chains).<br/>
+		</p>
+		<p>
+			The following diagram shows the way the complex relayer serves the lane in single direction. Everything
+			below may be applied to the opposite direction if you'll swap the Source and Target Chains.
+		</p>
+		<div class="mermaid">
+			sequenceDiagram
+			participant Source Chain
+			participant Complex Relayer
+			participant Target Chain
+
+			Note right of Source Chain: Finalized: 480, Target Finalized: 50, Sent Messages: 42, Confirmed Messages: 42
+			Note left of Target Chain: Finalized: 60, Source Finalized: 420, Received Messages: 42
+
+			Source Chain ->> Source Chain: someone Sends Message 43
+			Source Chain ->> Source Chain: Import and Finalize Block 481
+
+			Source Chain ->> Complex Relayer: notes new outbound message 43 at Source Chain Block 481
+			Note right of Complex Relayer: can't deliver message 43, Source Chain Block 481 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Source Chain Block 481
+
+			Source Chain ->> Complex Relayer: Read Finality Proof of Block 481
+			Complex Relayer ->> Target Chain: Submit Finality Proof of Block 481
+			Target Chain ->> Target Chain: Import and Finalize Block 61
+			Note left of Target Chain: Finalized: 61, Source Finalized: 481, Received Messages: 42
+
+			Source Chain ->> Complex Relayer: Read Proof of Message 43 at Block 481
+			Complex Relayer ->> Target Chain: Submit Proof of Message 43 at Block 481
+			Target Chain ->> Target Chain: Import and Finalize Block 62
+			Note left of Target Chain: Finalized: 62, Source Finalized: 481, Received Messages: { rewarded: 42, messages-relayer-account: [43] }
+
+			Target Chain ->> Complex Relayer: notes new unrewarded relayer at Target Chain Block 62
+			Note right of Complex Relayer: can't relay delivery confirmations because Target Chain Block 62 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Target Chain Block 62
+
+			Target Chain ->> Complex Relayer: Read Finality Proof of Block 62
+			Complex Relayer ->> Source Chain: Submit Finality Proof of Block 62
+			Source Chain ->> Source Chain: Import and Finalize Block 482
+			Note right of Source Chain: Finalized: 482, Target Finalized: 62, Confirmed Messages: 42
+
+			Target Chain ->> Complex Relayer: Read Proof of Message 43 Delivery at Block 62
+			Complex Relayer ->> Source Chain: Submit Proof of Message 43 Delivery at Block 612
+			Source Chain ->> Source Chain: rewards messages-relayer-account for delivering message [43]
+			Source Chain ->> Source Chain: prune delivered message 43 from runtime storage
+			Note right of Source Chain: Finalized: 482, Target Finalized: 61, Confirmed Messages: 43
+
+			Source Chain ->> Source Chain: someone Sends Message 44
+			Source Chain ->> Source Chain: Import and Finalize Block 483
+
+			Source Chain ->> Complex Relayer: notes new outbound message 44 at Source Chain Block 483 and new confirmed message 43
+			Note right of Complex Relayer: can't deliver message 44, Source Chain Block 483 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Source Chain Block 483
+
+			Source Chain ->> Complex Relayer: Read Finality Proof of Block 483
+			Complex Relayer ->> Target Chain: Submit Finality Proof of Block 483
+			Target Chain ->> Target Chain: Import and Finalize Block 63
+			Note left of Target Chain: Finalized: 63, Source Finalized: 483, Received Messages: { rewarded: 42, messages-relayer-account: [43] }
+
+			Source Chain ->> Complex Relayer: Read Proof of Message 44 and Proof of Message 43 reward at Block 483
+			Complex Relayer ->> Target Chain: Submit Proof of Message 44 and Proof of Message 43 reward at Block 483
+			Target Chain ->> Target Chain: Import and Finalize Block 64
+			Note left of Target Chain: Finalized: 64, Source Finalized: 483, Received Messages: { rewarded: 43, messages-relayer-account: [44] }-->
+		</div>
+		<script src="https://cdn.jsdelivr.net/npm/mermaid@8.8.4/dist/mermaid.min.js"></script>
+		<script>mermaid.initialize({startOnLoad: true})</script>
+	</body>
+</html>

docs/grandpa-finality-relay.html

@@ -0,0 +1,47 @@
+<!DOCTYPE html>
+<html>
+	<head>
+		<meta charset="utf-8">
+		<meta name="viewport" content="width=device-width">
+		<title>GRANDPA Finality Relay</title>
+	</head>
+	<body>
+		<h1>GRANDPA Finality Relay</h1>
+		<p>
+			Source Chain is running GRANDPA Finality Gadget. Bridge GRANDPA finality pallet is deployed at
+			Target Chain runtime. Relayer is configured to relay Source Chain finality to Target Chain.
+		</p>
+		<div class="mermaid">
+			sequenceDiagram
+			participant Source Chain
+			participant Relayer
+			participant Target Chain
+			Note left of Source Chain: Best: 500, Finalized: 480, Authorities Set Index: 42
+			Note right of Target Chain: Uninitialized
+
+			Source Chain ->> Relayer: Read Initialization Data
+			Relayer ->> Target Chain: Initialize Bridge GRANDPA Finality Pallet
+			Note right of Target Chain: Finalized: 480, Authorities Set Index: 42
+
+			Source Chain ->> Source Chain: Import Block 501
+			Source Chain ->> Source Chain: Import Block 502
+			Source Chain ->> Source Chain: Finalize Block 495
+			Source Chain ->> Relayer: Read Finality Proof of Block 495
+			Relayer ->> Target Chain: Finality Proof of Block 495
+			Note right of Target Chain: Finalized: 495, Authorities Set Index: 42
+
+			Source Chain ->> Source Chain: Import Block 503 that changes Authorities Set to 43
+			Source Chain ->> Source Chain: Finalize Block 500
+			Note left of Relayer: Relayer Misses Finality Notification for Block 500
+
+			Source Chain ->> Source Chain: Import Block 504
+			Source Chain ->> Source Chain: Finalize Mandatory Block 503
+			Source Chain ->> Source Chain: Finalize Block 504
+			Source Chain ->> Relayer: Read Finality Proof of Mandatory Block 503
+			Relayer ->> Target Chain: Finality Proof of Block 503
+			Note right of Target Chain: Finalized: 503, Authorities Set Index: 43
+		</div>
+		<script src="https://cdn.jsdelivr.net/npm/mermaid@8.8.4/dist/mermaid.min.js"></script>
+		<script>mermaid.initialize({startOnLoad: true})</script>
+	</body>
+</html>