Create a Dockerfile that is able to build and run Substrate

[poppyseedDev]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Motivation

As a developer interested in using the polkadot-sdk for Substrate development and making it accessible for more people, I find that the process of setting up and configuring the environment to be somewhat complex, especially for newcomers. Having a Dockerfile that can build and run Substrate would streamline the development process, making it more accessible and efficient.

Request

I propose the creation of a Dockerfile within the polkadot-sdk repository, which would handle the following:

1. Install all required dependencies for building Substrate.
2. Build polkadot-sdk.
3. Set up an environment to run the Substrate node.

This Dockerfile would abstract the complexity of environment setup and provide a straightforward path to getting Substrate running in a containerized environment.

Thank you for considering this request.

Solution

Something like this?

Dockerfile:

# Build stage for Substrate
FROM rust:latest as builder

# Install necessary build dependencies including cryptography package
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    clang \
    libclang-dev \
    protobuf-compiler \
    git \
    ca-certificates \
    llvm \
    libudev-dev \
    make \
    && rm -rf /var/lib/apt/lists/*

# Clone the GitHub repository
RUN git clone https://github.com/paritytech/polkadot-sdk.git

# Set the working directory to the cloned repository
WORKDIR /polkadot-sdk

# Install Rust toolchain and configure it
RUN rustup update && \
    rustup default stable && \
    rustup target add wasm32-unknown-unknown && \
    rustup update nightly && \
    rustup target add wasm32-unknown-unknown --toolchain nightly

# Build the project
RUN cargo build --release

# Use a smaller base image for the runtime environment
FROM debian:bookworm-slim as runtime

# Copy the built binary from the builder stage
COPY --from=builder /polkadot-sdk/target/release /usr/local/bin

# Expose necessary ports
EXPOSE 30333 9933 9944 9615

# Command to run the binary
CMD ["/usr/local/bin/substrate-node", "--dev", "--unsafe-rpc-external", "--rpc-cors", "all"]

Are you willing to help with this request?

Yes!

[ggwpez]

Why is it not using the parity linux image like here https://github.com/paritytech/polkadot-sdk/blob/master/substrate/docker/substrate_builder.Dockerfile ?
I think the problem with the existing docker files is that they are not really documented and not CI tested (so probably dont work). cc @chevdor (in case you have an alternative)

[xlc]

I suggest setup devcontainer integration https://code.visualstudio.com/docs/devcontainers/create-dev-container

https://github.com/AcalaNetwork/Acala/tree/master/.devcontainer

[poppyseedDev]

Why is it not using the parity linux image like here https://github.com/paritytech/polkadot-sdk/blob/master/substrate/docker/substrate_builder.Dockerfile ? I think the problem with the existing docker files is that they are not really documented and not CI tested (so probably dont work). cc @chevdor (in case you have an alternative)

I had some issues I didn't know how to solve while building with that image that's why I didn't include it.

The Dockerfile I provided in the description works well by building the image and is even able to connect to it, however I get a weird error in the end when trying to send the transaction that looks like this:

😞 Transaction Failed: RpcError: 1002: Verification Error: Runtime error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed WASM backtrace: error while executing at wasm backtrace: 0: 0x72893a - <unknown>!rust_begin_unwind 1: 0xdb82 - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e 2: 0x29d5f2 - <unknown>!TaggedTransactionQueue_validate_transaction: RuntimeApi, Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n 0: 0x72893a - <unknown>!rust_begin_unwind\n 1: 0xdb82 - <unknown>!core::pa…

[poppyseedDev]

Update this build runs fine:

Commands

docker build -t polkadot-node .

docker run --rm -it --name my-polkadot-node -p 30333:30333 -p 9933:9933 -p 9944:9944 substrate-node

Dockerfile

# Build stage for Substrate
FROM rust:latest as builder

# Install necessary build dependencies including cryptography package
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    clang \
    libclang-dev \
    protobuf-compiler \
    git \
    ca-certificates \
    llvm \
    libudev-dev \
    make \
    && rm -rf /var/lib/apt/lists/*

# Clone the GitHub repository
RUN git clone https://github.com/paritytech/polkadot-sdk.git

# Set the working directory to the cloned repository
WORKDIR /polkadot-sdk

# Checkout a specific release
RUN git config --global user.email "you@example.com"
RUN git config --global user.name "Your Name"
RUN git checkout release-polkadot-v1.3.0

# Install Rust toolchain and configure it
RUN rustup update && \
    rustup default stable && \
    rustup target add wasm32-unknown-unknown && \
    rustup update nightly && \
    rustup target add wasm32-unknown-unknown --toolchain nightly

# Build the project
RUN cargo build --release

# Use a smaller base image for the runtime environment
FROM debian:bookworm-slim as runtime

# Copy the built binary from the builder stage
COPY --from=builder /polkadot-sdk/target/release /usr/local/bin

# Expose necessary ports
EXPOSE 30333 9933 9944 9615

# Command to run the binary
CMD ["/usr/local/bin/polkadot", "--dev", "--unsafe-rpc-external", "--rpc-cors", "all"]

[poppyseedDev]

To sum up it is able to run ./target/release/polkadot build with release-polkadot-v1.3.0 or master branch, but it has issues when running ./target/release/substrate-node with the master branch.

And gets this weird error, when trying to do a transaction:

😞 Transaction Failed: RpcError: 1002: Verification Error: Runtime error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed WASM backtrace: error while executing at wasm backtrace: 0: 0x72893a - <unknown>!rust_begin_unwind 1: 0xdb82 - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e 2: 0x29d5f2 - <unknown>!TaggedTransactionQueue_validate_transaction: RuntimeApi, Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n 0: 0x72893a - <unknown>!rust_begin_unwind\n 1: 0xdb82 - <unknown>!core::pa…

[chevdor]

I think we used to have such an image but it was not used and probably got removed. It was also no dev container per say.
That being said, a dev container would make sense and it would require dropping some of the cleanup we curently do (mainly through using a clean image in a multi-stage dockerfile).

The closest image we have this the polkadot_builder.Dockerfile.

You can see it is based on ci-linux:production which is the best option to get started.
I would suggest NOT using FROM rust:latest as builder or you will need to redo lots of the work from ci-linux:production and maintain it. ci-linux:production is used by CI and to build the production images so you can rely on that one to work.

[ltfschoen]

ci-linux:production is used by CI and to build the production images so you can rely on that one to work.

if i use the latest ci-linux:production that is 10 months old and uses RUST_NIGHTLY=2023-05-23 on this line https://github.com/paritytech/scripts/blob/master/dockerfiles/ci-linux/Dockerfile#L7 to run the latest commit of polkadot-sdk it gives error

error: package jsonrpsee-types v0.22.5 cannot be built because it requires rustc 1.74.1 or newer, while the currently active rustc version is 1.70.0

so i think it should be updated to the latest Rust nightly that supports the polkadot-sdk.
we're trying to resolve it in this issue #2750