Skip to content

ci: migrate pinner to Filebase and proof-chain stack to Hardhat#341

Merged
noslav merged 8 commits into
mainfrom
migrate-pinner-to-filebase
May 22, 2026
Merged

ci: migrate pinner to Filebase and proof-chain stack to Hardhat#341
noslav merged 8 commits into
mainfrom
migrate-pinner-to-filebase

Conversation

@noslav
Copy link
Copy Markdown
Member

@noslav noslav commented May 22, 2026

Summary

This PR does two things, both gated on upstream changes that the bsp-agent CI was no longer compatible with:

  1. Pinner migration — bring bsp-agent in line with ewm-das#72 (Filebase replacement for the retired w3 CLI). The pinner is now FILEBASE_RPC_TOKEN-only, listens on :5080, and drops --w3-agent-key/--w3-delegation-proof-path.
  2. Proof-chain stack swap — replace trufflesuite/ganache-cli + ghcr.io/covalenthq/cqt-virtnet:latest (Truffle 5.4.21, last built Jan 2023, broken since solc-bin.ethereum.org started 301-redirecting) with us-docker.pkg.dev/covalent-project/network/cqt-staking:pectra-stable (hardhat fork-from-mainnet). This mirrors the proof-chain stack covalenthq/refiner already uses in CI today.

The two changes ship together because the bsp-agent CI workflow runs both as one stack — neither half is independently testable.

Confirmation — CI is green end-to-end

The agent runs all the way through: it consumes a replica from Redis, encodes it, pins via the new Filebase-backed pinner, submits the proof tx through the hardhat fork, and writes the binary file out locally.

ewm-das       | INFO  generated dag has root cid: bafybei…
bsp-agent     | Proof-chain tx hash: 0xbab629ab…b60199 for block-replica segment: 1-19727343-replica
bsp-agent     | File written successfully to: ./bin/block-ethereum/1-19727343-replica-0xbab629ab…

File-by-file

Pinner migration (Filebase)

  • docker-compose-ci.yml, docker-compose-local.yml, docker-compose-hardhat.yml — swap W3_* env + --w3-* pinner flags for FILEBASE_RPC_TOKEN; drop port 3001; fix $PROOF_OUT_HEX shell-escaping. docker-compose-hardhat.yml also moves off the legacy ipfs-pinner image to the unified ewm-das one.
  • .github/workflows/docker-image.yml — drop WEB3_JWT, W3_AGENT_KEY, W3_DELEGATION_FILE from the generated .env; add FILEBASE_RPC_TOKEN.
  • .envrc (gitignored, local-only) — drop W3_*/WEB3_JWT exports; add FILEBASE_RPC_TOKEN placeholder.

Proof-chain stack swap (Truffle → Hardhat)

  • docker-compose-ci.yml
    • Remove the node (ganache-cli) service and the cqt-virtnet service entirely.
    • Add eth-node (cqt-staking image as hardhat-node, forks from $ERIGON_NODE) and cqt-staking (proof-chain deployer, runs npm run docker:deploy).
    • Repoint the agent's links/depends_on to cqt-staking; flip --proof-chain-address from the ganache-deterministic 0xEa2ff902… to the hardhat-deployed 0xce44d283… (same address refiner uses).
    • Hardcode MB_RPC_URL=http://hardhat-node:8545/. The RPC_URL secret still pointed at ganache-cli, which doesn't exist on the network anymore; sourcing the URL from a stale secret was the cause of the post-swap "dial tcp: lookup ganache-cli" failure mid-PR.
  • .github/workflows/docker-image.yml — pass ERIGON_NODE into .env so the hardhat fork has an upstream RPC.

Required CI setup (out of band)

These need to exist as repo secrets on covalenthq/bsp-agent before CI passes:

  • FILEBASE_RPC_TOKEN — Filebase IPFS RPC API token, scoped per-bucket (Filebase console → bucket settings).
  • ERIGON_NODE — mainnet RPC URL for the hardhat fork. Use the same value refiner has.
  • PRIVATE_KEY — set for (BSP operator from cqt-staking's deployContractsAndAddOperators.js, derives to 0x15d34AAf54267DB7D7c367839AAf71A00a2C6A65, which is the whitelisted Block Specimen Producer on the hardhat-deployed BSP proof-chain). Well-known test mnemonic, no real funds.

Retired (can be deleted once this merges): WEB3_JWT, W3_AGENT_KEY, W3_DELEGATION_FILE.

Now unused

  • ghcr.io/covalenthq/cqt-virtnet:latest image — no remaining bsp-agent consumer.
  • ./ganache_data/ directory still in the repo tree but referenced by no compose file. Worth a follow-up git rm -r.

🤖 Generated with Claude Code

Upstream ewm-das #72 replaced the w3 CLI integration with a direct-HTTPS
Filebase client (FILEBASE_RPC_TOKEN, env-only). The pinner now listens
only on :5080 and no longer accepts --w3-agent-key / --w3-delegation-proof-path.

This change updates the three docker-compose files and the GitHub
Actions workflow so CI launches the new ewm-das image with the Filebase
token instead of the retired W3 credentials. docker-compose-hardhat.yml
also moves off the legacy ipfs-pinner image to the unified ewm-das one.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 15.31%. Comparing base (5e7ea7e) to head (d448432).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/storage/ipfs.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #341   +/-   ##
=======================================
  Coverage   15.31%   15.31%           
=======================================
  Files          52       52           
  Lines        6112     6112           
=======================================
  Hits          936      936           
  Misses       5131     5131           
  Partials       45       45           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

noslav and others added 4 commits May 21, 2026 18:06
The cqt-virtnet:latest image bundles Truffle 5.4.21 on node:14-alpine.
That version of Truffle hardcodes solc-bin.ethereum.org for its version-
list fetch; that hostname now 301-redirects to binaries.soliditylang.org
and Truffle 5.4 doesn't follow the redirect cleanly, so the version list
parses empty, "0.8.13" can't be resolved, and `truffle migrate` aborts
before the proof-chain contracts deploy — failing CI before the agent
even sees the BSP event.

Truffle's VersionRange.getCachedSolcByVersionRange() scans
~/.config/truffle/compilers/ for any cached soljson-v{X.Y.Z}*.js and
short-circuits the network fetch when one matches. So we wget the
official 0.8.13 binary directly from binaries.soliditylang.org into that
cache dir as a pre-step in the proof-chain container's entrypoint.

Long-term fix is in covalenthq/cqt-virtnet (bump Truffle to 5.11+, or
pin a docker-based solc). This is a no-rebuild workaround that unblocks
bsp-agent CI today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ing)

bsp-agent's CI was built around `trufflesuite/ganache-cli` +
`ghcr.io/covalenthq/cqt-virtnet:latest`, the latter pinned to Truffle
5.4.21 and last rebuilt January 2023. Truffle 5.4 hardcodes
solc-bin.ethereum.org for its version-list fetch; that hostname now
301-redirects to binaries.soliditylang.org, the redirect isn't followed
cleanly, and `truffle migrate` aborts before the proof-chain contracts
deploy. The previous commit worked around this with a wget-into-cache
hack; that commit is reverted earlier in this series.

This commit aligns bsp-agent CI with refiner CI, which already uses
the actively-maintained `cqt-staking:pectra-stable` image (hardhat
fork-from-mainnet + a `docker:deploy` npm script). The pattern is
already proven in covalenthq/refiner's docker-compose-ci.yml.

- `node` (ganache-cli) → `eth-node` (cqt-staking:pectra-stable as
  hardhat-node, port 8545, forking from $ERIGON_NODE).
- `cqt-virtnet` → `cqt-staking` (same image, container_name
  proof-chain, runs `npm run docker:deploy`).
- agent: `links` + `depends_on` repointed to `cqt-staking`; the
  proof-chain address flips from the ganache/truffle-deterministic
  0xEa2ff902dbeEECcc828757B881b343F9316752e5 to the
  hardhat-deployed 0xce44d283b806C62698285D83c2Ca3F1e42Eb7112
  (matches what docker-compose-hardhat.yml already uses locally and
  what refiner uses for bsp_proofchain_address).
- workflow: pass `ERIGON_NODE` into the generated .env so the hardhat
  fork has an upstream RPC to point at.

`ERIGON_NODE` will need to be added as a repo secret (refiner already
has it). `WEB3_JWT` and the `ghcr.io/covalenthq/cqt-virtnet` image
become unused after this and can be cleaned up downstream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After swapping the proof-chain stack from truffle/ganache to hardhat,
the agent was still resolving its RPC endpoint via the RPC_URL GitHub
secret (`MB_RPC_URL=\${RPC_URL}`). That secret still points at
`http://ganache-cli:8545` — a hostname that no longer exists on the
docker network — so the agent fails the chain-id lookup with
"dial tcp: lookup ganache-cli on 127.0.0.11:53: server misbehaving"
and aborts with "no chain id specified".

Hardcoding `MB_RPC_URL=http://hardhat-node:8545/` in the CI compose
matches the pattern refiner uses (refiner overrides proofchain_node to
the in-cluster hardhat hostname inside its docker config) and decouples
CI from drift in the RPC_URL secret value.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@noslav noslav changed the title ci(pinner): migrate from web3.storage to Filebase ci: migrate pinner to Filebase and proof-chain stack to Hardhat May 22, 2026
Aligns README prose with the Filebase + Hardhat migration in this PR:

- Intro paragraph: drop the "(also called cqt-virtnet)" parenthetical
  (the cqt-virtnet image is no longer used) and add the `ewm-das`
  pinner sidecar to the storage description.
- Environment section: replace the Pinata / Web3.Storage / IPFS_SERVICE_TOKEN
  paragraph with a note that the agent itself holds no storage
  credentials — uploads are delegated to the `ewm-das` pinner over
  HTTP, and FILEBASE_RPC_TOKEN lives in the pinner's own env, not
  bsp-agent's. Drop IPFS_SERVICE_TOKEN from the .envrc example and the
  direnv-output sample.
- Run example: update default --ipfs-pinner-server port from :3000 to
  :5080 (the port the new pinner binds).
- Flag docs: rewrite `--ipfs-pinner-server` description to refer to the
  ewm-das pinner sidecar by name and note its default listen address.
- Docker services list: replace `ganache-cli` and the bare "proof-chain"
  entry with `ewm-das`, `hardhat-node` (cqt-staking image, mainnet
  fork), and a more accurate proof-chain description that matches what
  docker-compose-ci.yml actually spins up after this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@noslav noslav requested a review from rogarcia May 22, 2026 02:32
noslav added 2 commits May 22, 2026 11:12
Temporary swap so CI exercises the freshly-built ewm-das image (which
replaces the internal IPFS gateway set). Will revert to :stable before
merge once the run goes green.
Internal constants were lagging the git tags (last touched at 1.9.2
while git is on v1.10.0). Bumping to 1.11.0 to match the upcoming
release that ships the Filebase pinner migration and the proof-chain
stack swap to Hardhat.
@noslav noslav force-pushed the migrate-pinner-to-filebase branch from d448432 to 88be0c2 Compare May 22, 2026 18:47
Copy link
Copy Markdown
Contributor

@rogarcia rogarcia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@noslav noslav merged commit 23a8bca into main May 22, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants