ci: migrate pinner to Filebase and proof-chain stack to Hardhat#341
Merged
Conversation
Upstream ewm-das #72 replaced the w3 CLI integration with a direct-HTTPS Filebase client (FILEBASE_RPC_TOKEN, env-only). The pinner now listens only on :5080 and no longer accepts --w3-agent-key / --w3-delegation-proof-path. This change updates the three docker-compose files and the GitHub Actions workflow so CI launches the new ewm-das image with the Filebase token instead of the retired W3 credentials. docker-compose-hardhat.yml also moves off the legacy ipfs-pinner image to the unified ewm-das one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #341 +/- ##
=======================================
Coverage 15.31% 15.31%
=======================================
Files 52 52
Lines 6112 6112
=======================================
Hits 936 936
Misses 5131 5131
Partials 45 45 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4 tasks
The cqt-virtnet:latest image bundles Truffle 5.4.21 on node:14-alpine.
That version of Truffle hardcodes solc-bin.ethereum.org for its version-
list fetch; that hostname now 301-redirects to binaries.soliditylang.org
and Truffle 5.4 doesn't follow the redirect cleanly, so the version list
parses empty, "0.8.13" can't be resolved, and `truffle migrate` aborts
before the proof-chain contracts deploy — failing CI before the agent
even sees the BSP event.
Truffle's VersionRange.getCachedSolcByVersionRange() scans
~/.config/truffle/compilers/ for any cached soljson-v{X.Y.Z}*.js and
short-circuits the network fetch when one matches. So we wget the
official 0.8.13 binary directly from binaries.soliditylang.org into that
cache dir as a pre-step in the proof-chain container's entrypoint.
Long-term fix is in covalenthq/cqt-virtnet (bump Truffle to 5.11+, or
pin a docker-based solc). This is a no-rebuild workaround that unblocks
bsp-agent CI today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit 9e48a5b.
…ing) bsp-agent's CI was built around `trufflesuite/ganache-cli` + `ghcr.io/covalenthq/cqt-virtnet:latest`, the latter pinned to Truffle 5.4.21 and last rebuilt January 2023. Truffle 5.4 hardcodes solc-bin.ethereum.org for its version-list fetch; that hostname now 301-redirects to binaries.soliditylang.org, the redirect isn't followed cleanly, and `truffle migrate` aborts before the proof-chain contracts deploy. The previous commit worked around this with a wget-into-cache hack; that commit is reverted earlier in this series. This commit aligns bsp-agent CI with refiner CI, which already uses the actively-maintained `cqt-staking:pectra-stable` image (hardhat fork-from-mainnet + a `docker:deploy` npm script). The pattern is already proven in covalenthq/refiner's docker-compose-ci.yml. - `node` (ganache-cli) → `eth-node` (cqt-staking:pectra-stable as hardhat-node, port 8545, forking from $ERIGON_NODE). - `cqt-virtnet` → `cqt-staking` (same image, container_name proof-chain, runs `npm run docker:deploy`). - agent: `links` + `depends_on` repointed to `cqt-staking`; the proof-chain address flips from the ganache/truffle-deterministic 0xEa2ff902dbeEECcc828757B881b343F9316752e5 to the hardhat-deployed 0xce44d283b806C62698285D83c2Ca3F1e42Eb7112 (matches what docker-compose-hardhat.yml already uses locally and what refiner uses for bsp_proofchain_address). - workflow: pass `ERIGON_NODE` into the generated .env so the hardhat fork has an upstream RPC to point at. `ERIGON_NODE` will need to be added as a repo secret (refiner already has it). `WEB3_JWT` and the `ghcr.io/covalenthq/cqt-virtnet` image become unused after this and can be cleaned up downstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After swapping the proof-chain stack from truffle/ganache to hardhat,
the agent was still resolving its RPC endpoint via the RPC_URL GitHub
secret (`MB_RPC_URL=\${RPC_URL}`). That secret still points at
`http://ganache-cli:8545` — a hostname that no longer exists on the
docker network — so the agent fails the chain-id lookup with
"dial tcp: lookup ganache-cli on 127.0.0.11:53: server misbehaving"
and aborts with "no chain id specified".
Hardcoding `MB_RPC_URL=http://hardhat-node:8545/` in the CI compose
matches the pattern refiner uses (refiner overrides proofchain_node to
the in-cluster hardhat hostname inside its docker config) and decouples
CI from drift in the RPC_URL secret value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aligns README prose with the Filebase + Hardhat migration in this PR: - Intro paragraph: drop the "(also called cqt-virtnet)" parenthetical (the cqt-virtnet image is no longer used) and add the `ewm-das` pinner sidecar to the storage description. - Environment section: replace the Pinata / Web3.Storage / IPFS_SERVICE_TOKEN paragraph with a note that the agent itself holds no storage credentials — uploads are delegated to the `ewm-das` pinner over HTTP, and FILEBASE_RPC_TOKEN lives in the pinner's own env, not bsp-agent's. Drop IPFS_SERVICE_TOKEN from the .envrc example and the direnv-output sample. - Run example: update default --ipfs-pinner-server port from :3000 to :5080 (the port the new pinner binds). - Flag docs: rewrite `--ipfs-pinner-server` description to refer to the ewm-das pinner sidecar by name and note its default listen address. - Docker services list: replace `ganache-cli` and the bare "proof-chain" entry with `ewm-das`, `hardhat-node` (cqt-staking image, mainnet fork), and a more accurate proof-chain description that matches what docker-compose-ci.yml actually spins up after this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Temporary swap so CI exercises the freshly-built ewm-das image (which replaces the internal IPFS gateway set). Will revert to :stable before merge once the run goes green.
Internal constants were lagging the git tags (last touched at 1.9.2 while git is on v1.10.0). Bumping to 1.11.0 to match the upcoming release that ships the Filebase pinner migration and the proof-chain stack swap to Hardhat.
d448432 to
88be0c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR does two things, both gated on upstream changes that the bsp-agent CI was no longer compatible with:
FILEBASE_RPC_TOKEN-only, listens on:5080, and drops--w3-agent-key/--w3-delegation-proof-path.trufflesuite/ganache-cli+ghcr.io/covalenthq/cqt-virtnet:latest(Truffle 5.4.21, last built Jan 2023, broken sincesolc-bin.ethereum.orgstarted 301-redirecting) withus-docker.pkg.dev/covalent-project/network/cqt-staking:pectra-stable(hardhat fork-from-mainnet). This mirrors the proof-chain stack covalenthq/refiner already uses in CI today.The two changes ship together because the bsp-agent CI workflow runs both as one stack — neither half is independently testable.
Confirmation — CI is green end-to-end
The agent runs all the way through: it consumes a replica from Redis, encodes it, pins via the new Filebase-backed pinner, submits the proof tx through the hardhat fork, and writes the binary file out locally.
File-by-file
Pinner migration (Filebase)
docker-compose-ci.yml,docker-compose-local.yml,docker-compose-hardhat.yml— swapW3_*env +--w3-*pinner flags forFILEBASE_RPC_TOKEN; drop port3001; fix$PROOF_OUT_HEXshell-escaping.docker-compose-hardhat.ymlalso moves off the legacyipfs-pinnerimage to the unifiedewm-dasone..github/workflows/docker-image.yml— dropWEB3_JWT,W3_AGENT_KEY,W3_DELEGATION_FILEfrom the generated.env; addFILEBASE_RPC_TOKEN..envrc(gitignored, local-only) — dropW3_*/WEB3_JWTexports; addFILEBASE_RPC_TOKENplaceholder.Proof-chain stack swap (Truffle → Hardhat)
docker-compose-ci.yml—node(ganache-cli) service and thecqt-virtnetservice entirely.eth-node(cqt-staking image ashardhat-node, forks from$ERIGON_NODE) andcqt-staking(proof-chain deployer, runsnpm run docker:deploy).agent'slinks/depends_ontocqt-staking; flip--proof-chain-addressfrom the ganache-deterministic0xEa2ff902…to the hardhat-deployed0xce44d283…(same address refiner uses).MB_RPC_URL=http://hardhat-node:8545/. TheRPC_URLsecret still pointed atganache-cli, which doesn't exist on the network anymore; sourcing the URL from a stale secret was the cause of the post-swap "dial tcp: lookup ganache-cli" failure mid-PR..github/workflows/docker-image.yml— passERIGON_NODEinto.envso the hardhat fork has an upstream RPC.Required CI setup (out of band)
These need to exist as repo secrets on
covalenthq/bsp-agentbefore CI passes:FILEBASE_RPC_TOKEN— Filebase IPFS RPC API token, scoped per-bucket (Filebase console → bucket settings).ERIGON_NODE— mainnet RPC URL for the hardhat fork. Use the same value refiner has.PRIVATE_KEY— set for (BSP operator fromcqt-staking'sdeployContractsAndAddOperators.js, derives to0x15d34AAf54267DB7D7c367839AAf71A00a2C6A65, which is the whitelisted Block Specimen Producer on the hardhat-deployed BSP proof-chain). Well-known test mnemonic, no real funds.Retired (can be deleted once this merges):
WEB3_JWT,W3_AGENT_KEY,W3_DELEGATION_FILE.Now unused
ghcr.io/covalenthq/cqt-virtnet:latestimage — no remaining bsp-agent consumer../ganache_data/directory still in the repo tree but referenced by no compose file. Worth a follow-upgit rm -r.🤖 Generated with Claude Code