Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible builds via Docker #799

Merged
merged 22 commits into from Aug 16, 2023
Merged

Conversation

capossele
Copy link
Contributor

@capossele capossele commented Aug 15, 2023

This PR extends the cargo risczero tool with a new build command.

It is based on a docker image built with this Dockerfile and pushed to the risczero Docker hub account.

Installation

To install the tool just run:

cargo install --path risc0/cargo-risczero/

Example usage

# Build the factors example
cargo risczero build --manifest-path examples/factors/methods/guest/Cargo.toml

This will generate an output similar to:

ELFs ready at:
./elfs/multiply/multiply - ImageID: 9ee1612b0a7e270f8df248e47dc85d9908ff4c1e7df42f398a65ca878b57a23d

Packages that generate multiple ELFs are supported as well:

# Build the chess example
cargo risczero build --manifest-path risc0/zkvm/methods/guest/Cargo.toml      

This will generate an output similar to:

ELFs ready at:
./elfs/risc0_zkvm_methods_guest/hello_commit - ImageID: eb12f9b97d8759327f651afeb09ae9a5713e7dbc428284d453b8cf56e8dadd5a
./elfs/risc0_zkvm_methods_guest/multi_test - ImageID: 761900e766a4ae1d8edcb2b49dc9aee54b94e42c9b0d6421cfb112314c4e3efc
./elfs/risc0_zkvm_methods_guest/slice_io - ImageID: 3f2ad1a2d500ab4ab927eebe241d872d3f598065b8987b182410cf01f350f74c

Resolves #755
Resolves #116

@flaub
Copy link
Member

flaub commented Aug 15, 2023

This is great! I think cargo risczero build (drop the -guest) is a better name.

@github-actions
Copy link

Benchmark for Linux-cuda

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for Linux-default

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for macOS-default

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for macOS-metal

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

@flaub
Copy link
Member

flaub commented Aug 15, 2023

It'd be great if we had a test for this somehow. I'm also interested in what instructions we need to have in docs for users on various platforms.

@capossele
Copy link
Contributor Author

Yeah I agree. I'm all open for testing suggestions. So far I came up with the following idea:

  • Build a few guests on Linux and on Mac and check that both result into same values. This requires having Docker installed on both. I'm not sure though if would be feasible to compare a GitHub job (on Linux) result to another job (on Mac)

@flaub
Copy link
Member

flaub commented Aug 15, 2023

What if we did a manual build locally on any platform, then we checked in the resulting IMAGE_ID. Then in CI, we just run the build and compare?

@capossele
Copy link
Contributor Author

Great idea! that would work! However we must be sure that the Cargo.lock used to do the local build is the same that gets checked in CI. I can try to put together a test for this 👍

Comment on lines 5 to 9
branches: [ main, 'release-*' ]
pull_request:
branches: [ main, 'release-*' ]

workflow_dispatch:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally I think this type of PR checker would live as job inside of the main.yml like: https://github.com/risc0/risc0/blob/main/.github/workflows/main.yml#L195

Any reason here to have it broken off?

clap = { version = "4.0", features = ["derive"] }
const_format = "0.2"
dirs = "5.0"
docker-generate = "0.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL about this crate! Super useful

@@ -0,0 +1,11 @@
# To build run: docker build -f Dockerfile.release -t risczero/risc0-guest-builder:v0.17 .
FROM ubuntu:20.04@sha256:3246518d9735254519e1b2ff35f95686e4a5011c90c85344c1f38df7bae9dd37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit 20.04 will EOL in 2024, vs 22.04 which is is EOL (base support tier) in 2026

Should we jump to 22.04 right now to extend the lifecycle of these images?

RUN curl --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused -fsSL 'https://sh.rustup.rs' | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN cargo install cargo-risczero
RUN cargo risczero install
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we update the remote toolchain, I suspect this could impact the image to change? @flaub is it possible to risczero install <VERSION> so we could optionally lock to a rust release?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think the next evolution of the install command is to have it specify version.

@capossele
Copy link
Contributor Author

The docker container already uses --locked as a manifest-path option to ensure that the Cargo.lock (of the guest package) is up-to-date. We can surely add a check even before starting the docker build to make sure the Cargo.lock file is indeed present

@github-actions
Copy link

Benchmark for Linux-cuda

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for Linux-default

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for macOS-default de0c3ba

Click to hide benchmark
Test Base PR %
fib/100/execute 2.7±0.11ms 2.7±0.18ms 0.00%
fib/100/prove 3.7±0.07s 3.6±0.06s -2.70%
fib/100/total 3.7±0.06s 3.7±0.06s 0.00%
fib/1000/execute 3.0±0.10ms 2.9±0.08ms -3.33%
fib/1000/prove 3.7±0.08s 3.7±0.03s 0.00%
fib/1000/total 3.7±0.07s 3.7±0.08s 0.00%
fib/10000/execute 5.2±0.11ms 5.0±0.15ms -3.85%
fib/10000/prove 15.2±0.11s 15.2±0.10s 0.00%
fib/10000/total 15.3±0.11s 15.2±0.13s -0.65%

Benchmark for macOS-metal de0c3ba

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.12ms 2.8±0.08ms 0.00%
fib/100/prove 853.9±2.77ms 849.6±5.76ms -0.50%
fib/100/total 875.6±6.51ms 872.0±4.45ms -0.41%
fib/1000/execute 2.9±0.15ms 2.8±0.08ms -3.45%
fib/1000/prove 874.4±4.14ms 870.6±3.85ms -0.43%
fib/1000/total 899.0±6.29ms 890.4±6.89ms -0.96%
fib/10000/execute 5.1±0.08ms 5.0±0.13ms -1.96%
fib/10000/prove 3.3±0.01s 3.3±0.01s 0.00%
fib/10000/total 3.3±0.01s 3.3±0.01s 0.00%

@heavypackets
Copy link
Contributor

My first thought it that we're using the term "reproducible build" in a non-standard way. For an example of historical context, the organization I've supported since my Docker days goes into the motivations: https://reproducible-builds.org.

A reproducible build implies: identical source code -> identical binary via a deterministic compiler, a guarantee that is portable across disparate environments into perpetuity. This is important for distributing applications that can be rebuilt anywhere with guaranteed security, a primary motivation behind reproducible builds in OSS.

This is not the contract we are implying with our tool. We are concerned about Image ID stability. As I understand, the Image ID ignores parts of the ELF that are unimportant to the program semantics, so binary comparison is unimportant in the zkVM security model. In our model, rustc could produce binaries who's binary hashes do not match but have the same Image ID -- that distinction might not be important in our VM security model, but that is not generalizable to meet the same bar implied by a "reproducible build" of a compiled program.

So, I'd recommend renaming this feature, preferably avoiding anything that implies a deterministic build. Then document it heavily, so people aren't confused what we are/aren't guaranteeing.

@mothran
Copy link
Contributor

mothran commented Aug 15, 2023

So, I'd recommend renaming this feature, preferably avoiding anything that implies a deterministic build. Then document it heavily, so people aren't confused what we are/aren't guaranteeing.

I am confused about this because we can emit a MemoryImage object reproduce-ably and deterministically using this system (ideally assuming we squash most of the entropy bugs). The ELF might not be, but you don't need a ELF, you can skip right to the memory image as your primary input on both the ZKVM and Bonsai.

@mothran
Copy link
Contributor

mothran commented Aug 15, 2023

So, I'd recommend renaming this feature, preferably avoiding anything that implies a deterministic build. Then document it heavily, so people aren't confused what we are/aren't guaranteeing.

I am confused about this because we can emit a MemoryImage object reproduce-ably and deterministically using this system (ideally assuming we squash most of the entropy bugs). The ELF might not be, but you don't need a ELF, you can skip right to the memory image as your primary input on both the ZKVM and Bonsai.

I just double checked and while the ImageID is derived from most members of the MemoryImage object, I think its possible to have the data on disk change without the ImageID change because of the PageTableInfo but I would wanna double check with @flaub first.

If that is the case then I think my above comment is probably not accurate.

@github-actions
Copy link

Benchmark for Linux-cuda b24512e

Click to hide benchmark
Test Base PR %
fib/100/execute 5.1±0.12ms 5.1±0.11ms 0.00%
fib/100/prove 755.7±4.22ms 753.3±3.45ms -0.32%
fib/100/total 757.0±2.15ms 754.1±3.86ms -0.38%
fib/1000/execute 5.6±0.11ms 5.6±0.12ms 0.00%
fib/1000/prove 783.3±3.26ms 782.2±4.28ms -0.14%
fib/1000/total 794.8±4.04ms 788.6±4.09ms -0.78%
fib/10000/execute 10.2±0.13ms 10.2±0.11ms 0.00%
fib/10000/prove 3.0±0.01s 3.0±0.00s 0.00%
fib/10000/total 3.0±0.00s 3.0±0.01s 0.00%

Benchmark for Linux-default b24512e

Click to hide benchmark
Test Base PR %
fib/100/execute 6.8±0.40ms 6.7±0.43ms -1.47%
fib/100/prove 5.1±0.03s 5.1±0.04s 0.00%
fib/100/total 5.1±0.03s 5.1±0.02s 0.00%
fib/1000/execute 7.9±0.62ms 7.8±0.54ms -1.27%
fib/1000/prove 5.1±0.02s 5.1±0.03s 0.00%
fib/1000/total 5.1±0.02s 5.1±0.04s 0.00%
fib/10000/execute 13.1±0.27ms 13.0±0.54ms -0.76%
fib/10000/prove 21.3±0.10s 21.2±0.11s -0.47%
fib/10000/total 21.2±0.12s 21.2±0.13s 0.00%

Benchmark for macOS-default b24512e

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.14ms 2.8±0.13ms 0.00%
fib/100/prove 3.7±0.07s 3.7±0.06s 0.00%
fib/100/total 3.7±0.07s 3.7±0.06s 0.00%
fib/1000/execute 3.0±0.10ms 2.9±0.06ms -3.33%
fib/1000/prove 3.7±0.05s 3.7±0.05s 0.00%
fib/1000/total 3.7±0.08s 3.6±0.08s -2.70%
fib/10000/execute 5.2±0.06ms 5.1±0.11ms -1.92%
fib/10000/prove 15.3±0.16s 15.3±0.13s 0.00%
fib/10000/total 15.3±0.10s 15.3±0.12s 0.00%

Benchmark for macOS-metal

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

@flaub
Copy link
Member

flaub commented Aug 16, 2023

The ImageID computation is a function that has a type like:

fn compute_image_id(used_elf_pages, page_size, page_table_addr, pc) -> ImageID

where:

  • used_elf_pages: the set of pages that are read by the ELF binary (this means that some ELF pages are not accessed and not needed)
  • page_size: The circuit is currently hard coded to 1024 bytes.
  • page_table_addr: The linear address where the page table exists in the runtime memory image. This is also hard coded in the circuit.
  • pc: Program counter, this is the starting address to begin execution.

If any of the parameters change, you get a different ImageID. The part that we're concerned with here for a "reproducible build" is just the used_elf_pages.

With this approach (using docker) or any other approach we attempt, the goal is to be able to ensure that if multiple users submit the same source code, they get identical used_elf_pages. Thus, to me this satisfies the requirements of a "reproducible build system". So I do think we imply "identical source code -> identical binary via a deterministic compiler". It just so happens that the "deterministic compiler" has to be wrapped in a docker image until rustc and cargo itself can satisfy this requirement.

reqwest = { version = "0.11", default-features = false, features = [
"blocking",
"json",
"rustls-tls",
"gzip",
] }
risc0-zkvm = { workspace = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to directly depend on risc0-binfmt here since all the types you're using from risc0-zkvm are just re-exports.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 👍

#[ignore] // requires Docker to be installed
fn test_reproducible_multiply_method() {
let multiply_test = Tester {
manifest_path: "examples/factors/methods/guest/Cargo.toml".to_string(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something a bit more persistent would be risc0/zkvm/methods/guest/Cargo.toml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not committing the Cargo.lock for that method though

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should. I'm thinking all guests should have their lock files committed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok added 👍

/// Overwrites if an ELF with the same name already exists.
fn build(&self) -> Result<()> {
Command::new("docker")
.args(["build", "--output=elfs/", "--target=binary", "."])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the output should land under target/, so maybe target/riscv-guest/riscv32im-risc0-zkvm-elf/docker?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

modified 👍

.gitignore Outdated
rust-project.json
target/
tmp/
elfs/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer the ELFs to land under target/ someplace

@heavypackets
Copy link
Contributor

  • used_elf_pages: the set of pages that are read by the ELF binary (this means that some ELF pages are not accessed
    If any of the parameters change, you get a different ImageID. The part that we're concerned with here for a "reproducible build" is just the used_elf_pages.

This highlights my concern. We aren't concerned about the resulting binary on disk being identical through separate builds and we aren't testing binary hashes in our build system. In this scenario, we could have different binaries on disk produce the same ID because parts are ignored -- that would be an example of a violation of reproducible builds as I've seen understood in OSS and in PL/compiler space.

For the Web3 example, the NEAR documentation linked in Slack does claim to build identical binaries: https://docs.near.org/sdk/rust/building/reproducible-builds

To verify your contract user can build it themselves and check that the binaries are identical.

In the NEAR case, two slightly different on disk but identical semantically contracts would not meet their definition of reproducibility.

We are claiming that some derivative of those compiled binaries + other inputs will be identical. That's a different definition, and one that requires its own terminology IMO. Or, at least, very careful explanation how/why our definition is different.

@flaub
Copy link
Member

flaub commented Aug 16, 2023

We are also coming up with a new Operating System, which is the zkvm. What I claim is that a MemoryImage is equivalent to what an ELF or PE does in other OSes. If we could have rustc emit a MemoryImage directly, that'd be ideal, and then we'd care about comparing MemoryImages bit for bit equivalence. It's possible that multiple ELFs could map to a single MemoryImage, but I don't understand why this matters. Think of it like when we do the conversion, that some bits are redacted. This is similar to the way some reproducible build steps work (they blank out or normalize bits that are entropic, like timestamps).

@flaub
Copy link
Member

flaub commented Aug 16, 2023

In fact, I think we should also tell users that they should check that their ELF binary is identical after using this docker build. I'm also fine with doing this comparison in our CI tests.

@github-actions
Copy link

Benchmark for Linux-cuda

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for Linux-default ff8635a

Click to hide benchmark
Test Base PR %
fib/100/execute 4.8±0.15ms 4.7±0.16ms -2.08%
fib/100/prove 2.6±0.21s 1673.8±7.15ms -35.62%
fib/100/total 2.3±0.14s 1678.8±14.96ms -27.01%
fib/1000/execute 5.2±0.16ms 5.2±0.09ms 0.00%
fib/1000/prove 3.0±1.63s 1699.8±4.99ms -43.34%
fib/1000/total 2.2±0.19s 1730.5±96.15ms -21.34%
fib/10000/execute 8.9±0.14ms 8.9±0.11ms 0.00%
fib/10000/prove 8.9±2.14s 6.4±0.08s -28.09%
fib/10000/total 7.3±0.35s 6.4±0.21s -12.33%

Benchmark for macOS-default ff8635a

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.12ms 2.7±0.14ms -3.57%
fib/100/prove 3.7±0.06s 3.7±0.06s 0.00%
fib/100/total 3.7±0.05s 3.7±0.06s 0.00%
fib/1000/execute 2.9±0.07ms 2.9±0.14ms 0.00%
fib/1000/prove 3.7±0.04s 3.7±0.05s 0.00%
fib/1000/total 3.8±0.04s 3.7±0.05s -2.63%
fib/10000/execute 5.2±0.13ms 5.1±0.07ms -1.92%
fib/10000/prove 15.3±0.10s 15.3±0.17s 0.00%
fib/10000/total 15.4±0.18s 15.3±0.07s -0.65%

Benchmark for macOS-metal ff8635a

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.06ms 2.7±0.05ms -3.57%
fib/100/prove 854.3±3.69ms 851.7±4.22ms -0.30%
fib/100/total 880.8±6.25ms 876.2±5.38ms -0.52%
fib/1000/execute 3.0±0.07ms 2.9±0.07ms -3.33%
fib/1000/prove 872.9±4.78ms 871.3±2.85ms -0.18%
fib/1000/total 898.2±7.78ms 895.6±4.81ms -0.29%
fib/10000/execute 5.1±0.12ms 4.9±0.05ms -3.92%
fib/10000/prove 3.3±0.02s 3.3±0.01s 0.00%
fib/10000/total 3.3±0.01s 3.3±0.01s 0.00%

@github-actions
Copy link

Benchmark for Linux-cuda 60989d8

Click to hide benchmark
Test Base PR %
fib/100/execute 5.1±0.12ms 5.1±0.11ms 0.00%
fib/100/prove 773.5±4.44ms 719.1±1.78ms -7.03%
fib/100/total 771.8±2.35ms 727.8±1.92ms -5.70%
fib/1000/execute 5.6±0.09ms 5.5±0.13ms -1.79%
fib/1000/prove 801.7±4.59ms 750.0±1.26ms -6.45%
fib/1000/total 805.2±2.75ms 758.7±2.52ms -5.77%
fib/10000/execute 10.2±0.14ms 10.1±0.11ms -0.98%
fib/10000/prove 3.3±0.01s 2.8±0.00s -15.15%
fib/10000/total 3.3±0.01s 2.8±0.00s -15.15%

Benchmark for Linux-default

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for macOS-default 60989d8

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.14ms 2.7±0.16ms -3.57%
fib/100/prove 3.7±0.05s 3.7±0.07s 0.00%
fib/100/total 3.7±0.06s 3.7±0.08s 0.00%
fib/1000/execute 3.0±0.17ms 2.8±0.11ms -6.67%
fib/1000/prove 3.7±0.05s 3.7±0.06s 0.00%
fib/1000/total 3.8±0.06s 3.7±0.05s -2.63%
fib/10000/execute 5.2±0.11ms 5.1±0.16ms -1.92%
fib/10000/prove 15.3±0.21s 15.2±0.16s -0.65%
fib/10000/total 15.4±0.18s 15.3±0.08s -0.65%

Benchmark for macOS-metal

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

@capossele capossele marked this pull request as ready for review August 16, 2023 15:24
@github-actions
Copy link

Benchmark for Linux-cuda 8dbd3c5

Click to hide benchmark
Test Base PR %
fib/100/execute 5.2±0.09ms 5.1±0.10ms -1.92%
fib/100/prove 765.9±5.85ms 730.3±2.99ms -4.65%
fib/100/total 764.8±3.30ms 734.8±2.16ms -3.92%
fib/1000/execute 5.6±0.09ms 5.6±0.11ms 0.00%
fib/1000/prove 793.0±3.66ms 757.4±1.52ms -4.49%
fib/1000/total 797.3±2.74ms 762.6±2.00ms -4.35%
fib/10000/execute 10.3±0.09ms 10.2±0.13ms -0.97%
fib/10000/prove 3.3±0.01s 2.8±0.00s -15.15%
fib/10000/total 3.3±0.01s 2.8±0.00s -15.15%

Benchmark for Linux-default

    <details open>
      <summary>Click to hide benchmark</summary>
      Benchmarks have changed between the two branches, unable to diff.
    </details>

Benchmark for macOS-default 8dbd3c5

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.15ms 2.7±0.13ms -3.57%
fib/100/prove 3.7±0.05s 3.7±0.07s 0.00%
fib/100/total 3.7±0.08s 3.6±0.07s -2.70%
fib/1000/execute 3.0±0.09ms 2.9±0.07ms -3.33%
fib/1000/prove 3.7±0.06s 3.7±0.07s 0.00%
fib/1000/total 3.7±0.06s 3.7±0.06s 0.00%
fib/10000/execute 5.2±0.12ms 5.0±0.08ms -3.85%
fib/10000/prove 15.3±0.22s 15.3±0.10s 0.00%
fib/10000/total 15.4±0.15s 15.3±0.11s -0.65%

Benchmark for macOS-metal 8dbd3c5

Click to hide benchmark
Test Base PR %
fib/100/execute 2.8±0.09ms 2.7±0.12ms -3.57%
fib/100/prove 854.9±3.72ms 848.9±5.25ms -0.70%
fib/100/total 875.8±7.12ms 875.1±7.21ms -0.08%
fib/1000/execute 2.9±0.07ms 2.9±0.09ms 0.00%
fib/1000/prove 872.7±3.25ms 869.2±5.48ms -0.40%
fib/1000/total 895.1±5.51ms 893.2±7.71ms -0.21%
fib/10000/execute 5.0±0.04ms 5.0±0.04ms 0.00%
fib/10000/prove 3.3±0.01s 3.3±0.01s 0.00%
fib/10000/total 3.3±0.01s 3.3±0.01s 0.00%

@capossele capossele merged commit 81578ba into main Aug 16, 2023
20 checks passed
@capossele capossele deleted the capossele/reproducible-builds branch August 16, 2023 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sending code or docker container to verifier Make ELF binaries build deterministically
7 participants