Skip to content
This repository has been archived by the owner on Apr 15, 2024. It is now read-only.

Developer test environment #613

Open
rach-id opened this issue Nov 22, 2023 · 2 comments
Open

Developer test environment #613

rach-id opened this issue Nov 22, 2023 · 2 comments
Labels
documentation Improvements or additions to documentation

Comments

@rach-id
Copy link
Member

rach-id commented Nov 22, 2023 •
edited
Loading

Summary

Create a Blobstream environment that rollup developers can use to speed up the process of getting started with Blobstream.

Context

Currently, we have the E2E test environment that can also be used as a development environment by running the whole cluster:

orchestrator-relayer/e2e/docker-compose.yml

Lines 1 to 315 in 0508c24

version: '3'
services:
# core0 always needs to start first for the E2E suite to work correctly
core0:
container_name: core0
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
ports:
- "9090:9090"
- "26657:26657"
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_core0.sh"
]
# EVM Address: 0x966e6f22781EF6a6A82BBB4DB3df8E225DfD9488
# key: da6ed55cb2894ac2c9c10209c09de8e8b9d109b910338d5bf3d747a7e1fc9eb9
volumes:
- ${PWD}/celestia-app/core0/config/priv_validator_key.json:/opt/config/priv_validator_key.json:ro
- ${PWD}/celestia-app/core0/config/node_key.json:/opt/config/node_key.json:ro
- ${PWD}/celestia-app/core0/keyring-test:/opt/keyring-test:ro
- ${PWD}/celestia-app/config.toml:/opt/config/config.toml:ro
- ${PWD}/celestia-app/app.toml:/opt/config/app.toml:ro
- ${PWD}/celestia-app/genesis_template.json:/opt/config/genesis_template.json:ro
- ${PWD}/celestia-app/genesis.json:/opt/config/genesis.json:rw
- ${PWD}/scripts/start_core0.sh:/opt/start_core0.sh:ro
core0-orch:
container_name: core0-orch
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core0
ports:
- "30000:30000"
environment:
- MONIKER=core0
- PRIVATE_KEY=da6ed55cb2894ac2c9c10209c09de8e8b9d109b910338d5bf3d747a7e1fc9eb9
- EVM_ACCOUNT=0x966e6f22781EF6a6A82BBB4DB3df8E225DfD9488
- CORE_RPC_HOST=core0
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core0
- CORE_GRPC_PORT=9090
# the other peers will consider core0 as a bootstrapper. So, it doesn't need to connect to any need during startup
- P2P_BOOTSTRAPPERS=
- P2P_IDENTITY=f7fd2be6b2abaa8519b0570d3a12e684a397456b0cc721198ebdd7e343cdd8f41813ef8aa857426c4e909c4fe19120a0f3f8c505286ca9d9249b5681475ec459
- P2P_LISTEN=/ip4/0.0.0.0/tcp/30000
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_orchestrator_after_validator_created.sh"
]
volumes:
- ${PWD}/celestia-app/core0/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_orchestrator_after_validator_created.sh:/opt/start_orchestrator_after_validator_created.sh:ro
core1:
container_name: core1
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core0
environment:
- MONIKER=core1
- CELESTIA_HOME=/opt
# key: 002ad18ca3def673345897b063bfa98d829a4d812dbd07f1938676828a82c4f9
- EVM_ACCOUNT=0x91DEd26b5f38B065FC0204c7929Da1b2A21877Ad
- AMOUNT=5000000000utia
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_node_and_create_validator.sh"
]
volumes:
- ${PWD}/celestia-app/core1/config/priv_validator_key.json:/opt/config/priv_validator_key.json:ro
- ${PWD}/celestia-app/core1/config/node_key.json:/opt/config/node_key.json:ro
- ${PWD}/celestia-app/core1/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_node_and_create_validator.sh:/opt/start_node_and_create_validator.sh:ro
- ${PWD}/celestia-app/config.toml:/opt/config/config.toml:ro
- ${PWD}/celestia-app/app.toml:/opt/config/app.toml:ro
- ${PWD}/celestia-app/genesis_template.json:/opt/config/genesis_template.json:ro
- ${PWD}/celestia-app/genesis.json:/opt/config/genesis.json:ro
core1-orch:
container_name: core1-orch
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core1
ports:
- "30001:30000"
environment:
- MONIKER=core1
- EVM_ACCOUNT=0x91DEd26b5f38B065FC0204c7929Da1b2A21877Ad
- PRIVATE_KEY=002ad18ca3def673345897b063bfa98d829a4d812dbd07f1938676828a82c4f9
- CORE_RPC_HOST=core1
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core1
- CORE_GRPC_PORT=9090
- P2P_BOOTSTRAPPERS=/dns/core0-orch/tcp/30000/p2p/12D3KooWBSMasWzRSRKXREhediFUwABNZwzJbkZcYz5rYr9Zdmfn
- P2P_LISTEN=/ip4/0.0.0.0/tcp/30000
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_orchestrator_after_validator_created.sh"
]
volumes:
- ${PWD}/celestia-app/core1/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_orchestrator_after_validator_created.sh:/opt/start_orchestrator_after_validator_created.sh:ro
core2:
container_name: core2
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core0
environment:
- MONIKER=core2
- CELESTIA_HOME=/opt
# key: 6adac8b5de0ba702ec8feab6d386a0c7334c6720b9174c02333700d431057af8
- EVM_ACCOUNT=0x3d22f0C38251ebdBE92e14BBF1bd2067F1C3b7D7
- AMOUNT=5000000000utia
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_node_and_create_validator.sh"
]
volumes:
- ${PWD}/celestia-app/core2/config/priv_validator_key.json:/opt/config/priv_validator_key.json:ro
- ${PWD}/celestia-app/core2/config/node_key.json:/opt/config/node_key.json:ro
- ${PWD}/celestia-app/core2/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_node_and_create_validator.sh:/opt/start_node_and_create_validator.sh:ro
- ${PWD}/celestia-app/config.toml:/opt/config/config.toml:ro
- ${PWD}/celestia-app/app.toml:/opt/config/app.toml:ro
- ${PWD}/celestia-app/genesis_template.json:/opt/config/genesis_template.json:ro
- ${PWD}/celestia-app/genesis.json:/opt/config/genesis.json:ro
core2-orch:
container_name: core2-orch
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core2
ports:
- "30002:30000"
environment:
- MONIKER=core2
- EVM_ACCOUNT=0x3d22f0C38251ebdBE92e14BBF1bd2067F1C3b7D7
- PRIVATE_KEY=6adac8b5de0ba702ec8feab6d386a0c7334c6720b9174c02333700d431057af8
- CORE_RPC_HOST=core2
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core2
- CORE_GRPC_PORT=9090
- P2P_BOOTSTRAPPERS=/dns/core0-orch/tcp/30000/p2p/12D3KooWBSMasWzRSRKXREhediFUwABNZwzJbkZcYz5rYr9Zdmfn
- P2P_LISTEN=/ip4/0.0.0.0/tcp/30000
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_orchestrator_after_validator_created.sh"
]
volumes:
- ${PWD}/celestia-app/core2/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_orchestrator_after_validator_created.sh:/opt/start_orchestrator_after_validator_created.sh:ro
core3:
container_name: core3
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core0
environment:
- MONIKER=core3
- CELESTIA_HOME=/opt
# key: 2f7f6763500dfb48b710a4c5c6c6a487b8aa6c7bc8b8a9b637a23f651f1c9b51
- EVM_ACCOUNT=0x3EE99606625E740D8b29C8570d855Eb387F3c790
- AMOUNT=5000000000utia
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_node_and_create_validator.sh"
]
volumes:
- ${PWD}/celestia-app/core3/config/priv_validator_key.json:/opt/config/priv_validator_key.json:ro
- ${PWD}/celestia-app/core3/config/node_key.json:/opt/config/node_key.json:ro
- ${PWD}/celestia-app/core3/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_node_and_create_validator.sh:/opt/start_node_and_create_validator.sh:ro
- ${PWD}/celestia-app/config.toml:/opt/config/config.toml:ro
- ${PWD}/celestia-app/app.toml:/opt/config/app.toml:ro
- ${PWD}/celestia-app/genesis_template.json:/opt/config/genesis_template.json:ro
- ${PWD}/celestia-app/genesis.json:/opt/config/genesis.json:ro
core3-orch:
container_name: core3-orch
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- core3
ports:
- "30003:30000"
environment:
- MONIKER=core3
- EVM_ACCOUNT=0x3EE99606625E740D8b29C8570d855Eb387F3c790
- PRIVATE_KEY=2f7f6763500dfb48b710a4c5c6c6a487b8aa6c7bc8b8a9b637a23f651f1c9b51
- CORE_RPC_HOST=core3
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core3
- CORE_GRPC_PORT=9090
- P2P_BOOTSTRAPPERS=/dns/core0-orch/tcp/30000/p2p/12D3KooWBSMasWzRSRKXREhediFUwABNZwzJbkZcYz5rYr9Zdmfn
- P2P_LISTEN=/ip4/0.0.0.0/tcp/30000
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_orchestrator_after_validator_created.sh"
]
volumes:
- ${PWD}/celestia-app/core3/keyring-test:/opt/keyring-test:ro
- ${PWD}/scripts/start_orchestrator_after_validator_created.sh:/opt/start_orchestrator_after_validator_created.sh:ro
ganache:
container_name: ganache
image: trufflesuite/ganache-cli
command: [
"--port=8545",
"--verbose",
"--host=0.0.0.0",
"--networkId=1234",
"--chainId=1234",
"--gasLimit=30000000",
"--account=\"0x0e9688e585562e828dcbd4f402d5eddf686f947fb6bf75894a85bf008b017401,0x90000000000000000000000\""
]
ports:
- "8545:8545"
deployer:
container_name: deployer
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- ganache
- core0
environment:
# By default, we don't want to run the deploy on each run.
- DEPLOY_NEW_CONTRACT=false
- EVM_CHAIN_ID=blobstream-e2e
- EVM_ACCOUNT=0x95359c3348e189ef7781546e6E13c80230fC9fB5
- PRIVATE_KEY=0e9688e585562e828dcbd4f402d5eddf686f947fb6bf75894a85bf008b017401
- CORE_RPC_HOST=core0
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core0
- CORE_GRPC_PORT=9090
- EVM_CHAIN_ID=1234
- EVM_ENDPOINT=http://ganache:8545
- STARTING_NONCE=earliest
entrypoint: [
"/bin/bash"
]
command: [
"/opt/deploy_blobstream_contract.sh"
]
volumes:
- ${PWD}/scripts/deploy_blobstream_contract.sh:/opt/deploy_blobstream_contract.sh:ro
relayer:
container_name: relayer
build:
context: ..
dockerfile: e2e/Dockerfile_e2e
depends_on:
- ganache
- core0
ports:
- "30004:30000"
environment:
- EVM_CHAIN_ID=blobstream-e2e
- EVM_ACCOUNT=0x95359c3348e189ef7781546e6E13c80230fC9fB5
- PRIVATE_KEY=0e9688e585562e828dcbd4f402d5eddf686f947fb6bf75894a85bf008b017401
- CORE_RPC_HOST=core0
- CORE_RPC_PORT=26657
- CORE_GRPC_HOST=core0
- CORE_GRPC_PORT=9090
- EVM_CHAIN_ID=1234
- EVM_ENDPOINT=http://ganache:8545
- STARTING_NONCE=latest
- P2P_BOOTSTRAPPERS=/dns/core0-orch/tcp/30000/p2p/12D3KooWBSMasWzRSRKXREhediFUwABNZwzJbkZcYz5rYr9Zdmfn
- P2P_LISTEN=/ip4/0.0.0.0/tcp/30000
# set the following environment variable to some value
# if you want to relay to an existing Blobstream contract
# - BLOBSTREAM_CONTRACT=0x123
entrypoint: [
"/bin/bash"
]
command: [
"/opt/start_relayer.sh"
]
volumes:
- ${PWD}/scripts/start_relayer.sh:/opt/start_relayer.sh:ro
- ${PWD}/scripts/deploy_blobstream_contract.sh:/opt/deploy_blobstream_contract.sh:ro 

cd e2e
docker-compose up -d

This environment contains the following resources:

  • 4 Celestia validators
  • 4 orchestrators
  • 1 relayer
  • 1 Ethereum testnet: ganache

And when you run it, it will start the Celestia validators, these validators will create attestations. The orchestrators will pick up those attestations, sign them, and then push the signatures to the Blobstream P2P network (DHT). Finally, the relayer will query those signatures from the P2P network, and submit them to the Blobstream contract deployed in the ganache instance.

This can be used by rollup developers as a cluster containing a Celestia testnet with Blobstream deployed on it, and they can run their rollup sequencer next to it and test it.

Proposal

We can create an even better environment developer-oriented. This environment would imitate most of the configuration used in the E2E one but will add a simple sequencer to it, and verification logic.

The topology can look as follows:

  • 1 Celestia validator
  • 1 orchestrator
  • 1 relayer
  • 1 Ethereum testnet: ganache
  • 1 sequencer: which would receive transactions from the rollup users
  • 1 verifier: which would verify that the rollup data was posted on Celestia

Sequencer

The sequencer can be thought of as a simple for loop that does the following:

  • Create a rollup block containing some transactions:
    • The transactions can be as simple as (from, to, amount)
  • Create a blob transaction containing the block data and submit it to Celestia
  • Create the block header containing the blob commitment, the height, and the blob transaction hash (we add it for now since we still need to implement querying the proof using Celestia node only using the blob commitment)
  • Send the block header to the verifier:
    • The verifier can be as simple as a function in the sequencer so that we don't have to implement an API or some event bus to send the header and block to verifier.

Verifier

  • The verifier waits for the attestation that covers the block where the sequencer transaction was added until the state machine creates it, the orchestrator signs it and the relayer sends it to the Blobstream contract
  • The verifier creates a proof for the rollup block data and verifies it against the Blobstream contract.

We can even think of adding malicious events where fraud proofs will be emitted, but that can be done afterward.

Conclusion

Having a test environment like this one would help rollup developers to see the E2E flow that they would need to implement, and also give them an idea of the creation of the proofs in a project that works and where they can make simple changes and interact with it.
@rach-id rach-id added the documentation Improvements or additions to documentation label Nov 22, 2023
@rach-id
Copy link
Member Author

rach-id commented Nov 22, 2023

I think it would be best if we could prioritize implementing the proofs in Celestia-node so that we can get proofs using a light client without connecting with an RPC. Also, it would be good to implement the subtree roots verification in Blobstream contract so that users don't have to send the data to see if it was posted or not.

cc @evan-forbes

@evan-forbes
Copy link
Member

prioritizing other things first definitely makes sense to me, thanks for documenting this!

@rach-id rach-id mentioned this issue Nov 22, 2023
Merged
5 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rach-id @evan-forbes