Add container build backend and build verify command#2525
Add container build backend and build verify command#2525leighmcculloch wants to merge 66 commits intomainfrom
Conversation
|
Each meta field needs a very detailed specification of exactly what valid values are expected, and the format(s) those values can take. For example, depending on how you install the Homebrew: Cargo Cargo git install: Without a precise spec for each field, downstream tooling (verifiers, registries, indexers) can't reliably parse or validate these values. |
|
Currently only the That works, and the host-side stellar-cli version is captured in the wasm via the A way to get the best of both: Embed a Dockerfile in the stellar-cli that, at build time, layers on top of whatever rust base image is requested:
Both the rust version and the stellar-cli version are specified at build (and verify) time, the image gets built locally, and the entire build pipeline runs inside it. The user on the host doesn't need to install a matching stellar-cli to verify a build produced by a different version — that version is installed into the image instead. This avoids the supply-chain cost of us owning and publishing a bespoke stellar build image, keeps the rust base image flexible, and still lets the image digest capture the whole pipeline. It also resolves the It also keeps the door open for SDF or someone else to host prebuilt images later as a convenience — but we don't have to figure that out for the first iteration. The embedded-Dockerfile approach defers that decision without foreclosing it. |
|
Opened an issue about the cliver inconsistency here: |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
Opened an issue about dbus creating problems with using the image for the build for the verification step: |
|
@leighmcculloch I just tried this, but I'm getting a warning, even though there are no unstaged files. $ git status
On branch main
nothing to commit, working tree clean
$ stellar contract build --backend docker
⚠️ git working tree has uncommitted changes; source_repo/source_rev/bldopt_* not embedded in contract metadata. Commit changes for a reproducible build. |
What
Add
--backend docker[=<image>]tostellar contract build(anddeploy/upload) that runs the entire build pipeline inside a container whose entrypoint isstellar. Add astellar contract build verifysubcommand that reads everything it needs from the wasm's metadata, rebuilds, and reports which (if any) rebuilt artifact is byte-identical to the original. Add a mainnet warning onstellar contract deploywhen the wasm is missing the meta entries needed for independent verification.Why
Contract builds vary across host OS, architecture, and toolchain, preventing third parties from independently confirming a deployed contract was built from given source. Pinning the build to a docker image plus the rust toolchain version makes builds reproducible, recording the source repo + commit + per-package build options lets verifiers rebuild the exact same artifact, and the new
verifysubcommand automates the rebuild-and-compare check.Closes #2506.
How it works
Three parts: build-time recording, deploy-time warning, and verify-time reproduction.
Build
For all backends (including
local), the build:originremote, embedssource_repo(URL canonicalized tohttps://…),source_rev(full HEAD SHA), and per-package build options (bldopt_manifest_pathrelative to git root,bldopt_package,bldopt_profile, optionalbldopt_optimize). The manifest path is auto-inserted whether or not--manifest-pathwas passed on the CLI.source_repo/source_rev/bldopt_*.For
--backend docker, additionally:docker.io/stellar/stellar-cli@sha256:cb2fc3116a6ace37a77ca6bb88afb4bee57fc746cd556a4373f2c3ee95d4e917— pinned by digest so the recordedbldimgis reproducible from day one and we sidestep the longstanding Apple Silicon docker quirk where pulling a multi-arch tag with--platform=linux/amd64leavesRepoDigestsempty after pull.<git_root or workspace_root>→/source(rw, source — also where cargo writes its target dir, shared with the host)~/.cargo/registry→/usr/local/cargo/registry(rw, cached crate downloads)stellardirectly, bypassing the official image'sentrypoint.sh(which launchesdbus+gnome-keyringand trips when running under a host UID with no/etc/passwdentry — see Docker image's entrypoint dbus init fails when run as non-root UID #2543).contract builddoesn't use the keyring, so the wrapper is irrelevant here.stellar contract build --manifest-path /source/<rel> --profile <p> --locked --meta bldimg=<digest> [forwarded args]inside the container. The args use only flags that exist in publishedstellar/stellar-cliimages today; no new flags are added, and--backend localis deliberately not passed (it's a flag added in this PR and isn't recognized by published images).wasm-optitself; the host only orchestrates and copies outputs to--out-dirif requested.The wasm's
contractmetav0custom section is populated with up to nine entries:cliver26.0.0#abc1234…(CLI version + git rev)^\d+\.\d+\.\d+(-[A-Za-z0-9.+-]+)?#([0-9a-f]{40}(-dirty)?)?$bldimgdocker.io/stellar/stellar-cli@sha256:…^[^@\s]+@sha256:[0-9a-f]{64}$--backend docker)rsver1.83.0(resolved rustc version)^\d+\.\d+\.\d+(-[A-Za-z0-9.+-]+)?$source_repohttps://github.com/user/repo(clean repo's origin)^https?://\S+$source_rev^[0-9a-f]{40}$bldopt_manifest_pathcontracts/foo/Cargo.toml(relative to git)^([^/\s]+/)*Cargo\.toml$bldopt_package^[A-Za-z][A-Za-z0-9_-]*$bldopt_profilerelease)^[A-Za-z][A-Za-z0-9_-]*$bldopt_optimizetrue(only present when--optimizewas used)^true$The presence of
bldimgis what distinguishes a docker build from a local one — there's no separatebldbkdfield. For full reproducibility from day one, pin to a specific image with--backend docker=<name>@sha256:…and commit before building.--backendand--docker-hostare also exposed onstellar contract deployandstellar contract upload(which auto-build when no--wasm/--wasm-hashis given), so the same flags work end-to-end.Deploy
stellar contract deployagainst mainnet now warns when the wasm is missing any ofcliver,bldimg,rsver,source_repo,source_rev,bldopt_manifest_path,bldopt_package,bldopt_profile:The check is mainnet-only (matches network passphrase against
Public Global Stellar Network ; September 2015); on testnet/futurenet/local the wasm deploys silently.Verify
verifyis a subcommand ofbuild— it lives atstellar contract build verify, and works on multi-contract workspaces by rebuilding and finding the match.contract info).cliver,bldimg(optional),rsver, andbldopt_*(optional, best-effort) from the wasm's meta. Missingbldopt_*entries trigger a warning rather than an error and the build falls back to its defaults — verify still runs, just with the caveat that the rebuild may not be reproducible.bldimgpresent →Backend::Docker { image: bldimg }. The image's pinned digest pulls the same in-container cli that produced the original.bldimgabsent →Backend::Local. Best-effort rebuild on the host.rsverto the rebuild asRUSTUP_TOOLCHAIN(in-container) orcargo +<rsver>(local). For docker the toolchain inside the image is fixed by whoever built it; passingRUSTUP_TOOLCHAINlets rustup-managed cargo switch toolchains if the image carries multiple ones.bldopt_manifest_pathagainst the cwd's git top-level (viagit rev-parse --show-toplevel) so verify works from anywhere inside the checkout.The user is responsible for checking out the matching commit before running verify; verify rebuilds from the working tree. (
source_repoandsource_revare embedded in meta to help users find the right commit, but verify itself doesn't clone — that would add a separate trust path.)End-to-end example
The host CLI's version is irrelevant for verifying a docker-built wasm — whatever cli is in the image is what built (and rebuilds) the wasm.
Notes
/var/run/docker.sock, or whatever--docker-host/DOCKER_HOSTpoints at). Sameconnect_to_dockerhelper used bystellar container start/stop/logs, with the same Docker Desktop fallback ($HOME/.docker/run/docker.sock). No shell-out to thedockerCLI. A podman socket exposing the Docker API would also work (untested).--backend docker(no=...) defaults todocker.io/stellar/stellar-cli@sha256:cb2fc3..., notstellar/stellar-cli:latest. Recording a digest immediately makes builds reproducible day one and avoids the Apple SiliconRepoDigests-after-cross-platform-pull quirk. Bumping the default is a single-line const change inbuild.rs(see comments there for the recipe). Users who want a different image specify--backend docker=....stellar/stellar-cliimage's entrypoint runsentrypoint.sh, which launchesdbus+gnome-keyring. That setup fails when the container runs as a host UID without an/etc/passwdentry — see Docker image's entrypoint dbus init fails when run as non-root UID #2543. We override the entrypoint to point straight at thestellarbinary, which is fine becausecontract builddoesn't touch the keyring.~/.cargo/registrylets the container reuse crate downloads the host already has.wasm32v1-nonepre-installed for its default toolchain; ifRUSTUP_TOOLCHAINselects a different one (verify on a wasm built with another rust version), the cli/cargo handle target installation themselves.RUSTUP_TOOLCHAIN=<rsver>inside the container (andcargo +<rsver>for local rebuilds) so the rust version matches whatever the original build used.bldimgis normalized to<registry>/<path>@sha256:<digest>(e.g.stellar/stellar-cli:latest→docker.io/stellar/stellar-cli@sha256:…) so verify can resolve it without relying on the local registry config.source_repois normalized tohttps://…form (e.g.git@github.com:user/repo.git→https://github.com/user/repo).bldopt_manifest_pathis recorded relative to the git repo root regardless of whether--manifest-pathwas passed on the CLI. Verify resolves it against the cwd's git top-level so the command works from anywhere inside the checkout.stellar contract buildinside the image with only flags that exist in publishedstellar/stellar-cliimages today (--manifest-path,--profile,--locked,--meta,--package,--features,--all-features,--no-default-features,--optimize).bldimgis forwarded via--meta bldimg=<digest>, not a new flag.bldbkdfield: presence ofbldimgis the only signal needed to distinguish a docker build from a local one.docker container prune.Performance/runtime caveats
Building inside an
amd64container on a non-amd64 host (Apple Silicon, Linux/arm64) runs under emulation. For small contracts the difference is negligible; for workspaces with heavy dep trees the emulated build can be substantially slower than a native host build. Container runtimes that don't ship qemu/binfmt support won't run amd64 containers on arm64 hosts at all. See #2506 (comment).Related issues
stellar versionandclivermeta #2535 — normalizecliverrendering across install paths so it's a single shape.stellar/stellar-cliimage'sentrypoint.shfails under non-root UIDs, motivating the entrypoint override here.Status
This is an experiment in validating the ideas in #2506. May or may not be destined for merging — at this moment it's an experiment in validating the approach.