Skip to content

workspace: fold the offchain crates into the root workspace - #4255

Merged
bgm-malbeclabs merged 3 commits into
mainfrom
migrate/merge-workspaces
Sep 2, 2026
Merged

workspace: fold the offchain crates into the root workspace#4255
bgm-malbeclabs merged 3 commits into
mainfrom
migrate/merge-workspaces

Conversation

@bgm-malbeclabs

@bgm-malbeclabs bgm-malbeclabs commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Step 4 of the monorepo migration, per docs/superpowers/specs/2026-08-27-monorepo-migration-design.md. Stacked on #4245 and based on that branch, so the diff shown here is step 4 alone. It retargets as the stack lands.

The offchain tree stops being its own workspace. offchain/Cargo.toml, offchain/Cargo.lock and offchain/rust-toolchain.toml are deleted, its 15 crates become root members, and everything resolves against one lockfile on the 1.97.1 toolchain. solana/ stays excluded with its own three files, because 62 of the 96 crates in the programs' build closure resolve differently here and that would change the compiled bytes behind solana/programs/sha256sums_*.txt.

Summary

  • One workspace, one lockfile. The ten path dependencies from offchain: replace the ten git dependencies with path dependencies #4245 now resolve inside a single workspace rather than across two. No git+ source for this repo or for doublezero-solana remains in Cargo.lock; the only git dependency left is network-shapley, which lives in another organization.
  • Two inherited keys had to be stated per crate. All 15 crates are edition 2024 and this workspace is 2021, so an inherited edition would have moved them to 2021, where they do not build. Four of them also inherited the version, which would have taken them from 0.0.1 to 0.38.0. The spec called the first trap; the second turned up here.
  • bincode and reqwest are declared per crate in the offchain tree. This repo is on bincode 2, whose API differs, and on reqwest with its default TLS backend, while the offchain crates use bincode 1 and rustls with the OS root store. Unifying either is a code change, not a manifest change, and the reqwest one would move ip_proof — a trust boundary — onto a different TLS stack. Everything else reconciles to the tighter requirement or the union of features.
  • The e2e sentinel binary becomes dz-e2e-sentinel (D4). Offchain ships a deb named doublezero-sentinel, and two members of one workspace cannot write the same file into target/release. The two images follow the new name. Both binaries now build side by side.
  • The e2e base image names the four binaries it serves instead of building --workspace, which would now compile arrow, parquet and the AWS SDK on the way to a client and a sentinel.
  • 25 clippy fixes in the offchain crates. 21 are unreadable_literal, which this repo's lint line adds and offchain's did not. The rest: a match that collapses into guards, three redundant .into_iter() calls in zip, a loop counter, and a sort_by that is a sort_by_key.
  • The revdist fixture generator reaches its two solana crates by path, matching the other three generators. Its lockfile had drifted to solana#121 and the fixtures still regenerate byte-identical.

Two deliberate departures from the spec

  • Offchain's release profile (lto = true, codegen-units = 1) is not carried over. Cargo profiles are workspace-wide, so keeping it would apply full link-time optimization to every release build here, including the ones e2e waits on. The offchain release binaries lose that optimization. Accepted, not deferred: step 5 has no profile work to do. A custom release-lto profile would be the way back, and it would need care, because goreleaser's rust builder looks for target/<triple>/release.
  • Solana's >=2,<=3 ranges are left as ranges. The spec's instruction to pin them was written when the programs were joining the workspace. D2 holds them back and doublezero-shreds still consumes them by git, so the ranges still earn their keep for an external consumer.

Testing Verification

  • cargo clippy --workspace --all-targets with the Makefile's exclusions and -Dclippy::all -Dclippy::unreadable_literal -Dwarnings: clean.
  • cargo test --workspace --all-features with the same exclusions: 439 passed. The one failure is ip_proof::tests::probe_skips_a_local_address_that_cannot_reach_the_destination, in a root crate this branch does not touch. It asserts that the kernel rejects a loopback source toward a public address; the syscall reproduces on darwin as accepted for UDP, so the probe returns Bind. Linux only.
  • The step 1 reward goldens pass under the merged lockfile. test_aggregated_shapley_output_matches_golden and test_per_city_shapley_output_matches_golden both green, which is the risk the spec singles out as the serious one: contributor reward figures changing silently in a new lockfile.
  • Program unit tests for all five ledger programs: 420 passed. doublezero accounts compat check across all three environments: exit 0. Deployed program bytes are unaffected either way, since cargo build-sbf takes features from each crate's own defaults rather than from workspace dependency entries.
  • cd solana && cargo check --locked --workspace --all-targets: exit 0, and that lockfile needs no change.
  • cargo build --workspace: exit 0. cargo +nightly fmt --all --check: clean, including three import blocks in solana/ that cargo fmt --all reaches through the new path dependencies.
  • Duplicate dependency groups go from 140 to 176. Every new group comes from the offchain dependency set arriving in this lockfile, which the merge makes visible rather than causes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR is step 4 of the monorepo migration. It removes the standalone offchain/ Rust workspace and folds its crates into the root workspace so the repo builds against a single root lockfile and toolchain, while keeping solana/ excluded with its own lockfile to preserve reproducible program bytes.

Changes:

  • Add the 15 offchain crates as root workspace members, and move their shared dependencies into root [workspace.dependencies].
  • Update offchain crate manifests to state edition = "2024" and keep bincode and reqwest pinned per-crate to avoid API and TLS-backend changes.
  • Rename the e2e sentinel binary to dz-e2e-sentinel and update e2e Docker builds to compile only the required binaries; update the revdist fixture generator to use path dependencies into solana/.
File summaries
File Description
solana/programs/revenue-distribution/tests/initialize_distribution_test.rs Import cleanup to match formatting after workspace changes.
solana/programs/revenue-distribution/src/state/distribution.rs Import cleanup to match formatting after workspace changes.
solana/programs/revenue-distribution/src/state/contributor_rewards/mod.rs Import cleanup to match formatting after workspace changes.
sdk/revdist/testdata/fixtures/generate-fixtures/Cargo.toml Switch solana dependencies from git to local path dependencies.
sdk/revdist/testdata/fixtures/generate-fixtures/Cargo.lock Remove git sources for the solana crates after path dependency switch.
offchain/scheduler/native/scheduler_doublezero/Cargo.toml Move to edition 2024 and pin reqwest TLS backend; now part of root workspace.
offchain/rust-toolchain.toml Delete offchain-scoped toolchain so offchain crates use the repo toolchain.
offchain/crates/validator-debt/src/worker/mod.rs Minor clippy cleanup (remove redundant references in format args).
offchain/crates/validator-debt/src/validator_debt.rs Clippy unreadable_literal fixes in tests.
offchain/crates/validator-debt/src/rewards.rs Clippy unreadable_literal fixes in tests.
offchain/crates/validator-debt/src/jito.rs Clippy unreadable_literal fixes in tests.
offchain/crates/validator-debt/src/inflation.rs Clippy unreadable_literal fixes in tests.
offchain/crates/validator-debt/src/block.rs Clippy unreadable_literal fixes in tests.
offchain/crates/validator-debt/Cargo.toml Set edition 2024; pin bincode/reqwest per-crate to avoid API/TLS changes.
offchain/crates/solana-sdk/Cargo.toml Set explicit version and edition (avoid inheriting root workspace values).
offchain/crates/solana-interface/sol-conversion/Cargo.toml Set edition 2024 explicitly.
offchain/crates/solana-fork/Cargo.toml Set edition 2024 explicitly.
offchain/crates/solana-client-tools/Cargo.toml Set explicit version and edition; pin bincode 1 per-crate.
offchain/crates/solana-cli/src/command/shreds/validator_client_rewards/show.rs Clippy cleanup (remove redundant .into_iter() in zip).
offchain/crates/solana-cli/src/command/shreds/validator_client_rewards/init_holding.rs Clippy cleanup (remove redundant .into_iter() in zip).
offchain/crates/solana-cli/src/command/shreds/list.rs Clippy cleanup (remove redundant .into_iter() in zip).
offchain/crates/solana-cli/src/command/revenue_distribution/fetch/distribution.rs Simplify match into guards (clippy cleanup).
offchain/crates/solana-cli/Cargo.toml Set edition 2024; pin reqwest per-crate.
offchain/crates/solana-admin-cli/sol-conversion/Cargo.toml Set edition 2024 explicitly.
offchain/crates/solana-admin-cli/revenue-distribution/Cargo.toml Set edition 2024 explicitly.
offchain/crates/solana-admin-cli/passport/Cargo.toml Set edition 2024 explicitly.
offchain/crates/slack-notifier/Cargo.toml Set edition 2024; pin reqwest per-crate.
offchain/crates/sentinel/Cargo.toml Set edition 2024; pin bincode/reqwest per-crate.
offchain/crates/scheduled-command/Cargo.toml Set explicit version and edition (avoid inheriting root workspace values).
offchain/crates/passport-cli/Cargo.toml Set explicit version and edition (avoid inheriting root workspace values).
offchain/crates/contributor-rewards/tests/test_s3_storage.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/tests/test_pvt_links.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/tests/test_pub_links.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/tests/test_historical_epoch.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/tests/common/mod.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/src/ingestor/inet_accumulator.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/src/ingestor/epoch.rs Clippy unreadable_literal fixes in tests.
offchain/crates/contributor-rewards/src/ingestor/demand.rs Import cleanup and use sort_by_key with Reverse.
offchain/crates/contributor-rewards/src/cli/inspect.rs Simplify loop counter logic using (1u32..).zip(cities).
offchain/crates/contributor-rewards/src/calculator/shapley/handler.rs Clippy unreadable_literal fix in tests.
offchain/crates/contributor-rewards/src/calculator/constants.rs Clippy unreadable_literal fix for a constant.
offchain/crates/contributor-rewards/Cargo.toml Set edition 2024 explicitly.
offchain/Cargo.toml Delete the offchain workspace manifest now that crates are root members.
e2e/docker/sentinel/Dockerfile Use the renamed dz-e2e-sentinel binary.
e2e/docker/base.dockerfile Build only the four required binaries instead of --workspace.
crates/sentinel/Cargo.toml Rename the e2e sentinel binary to avoid artifact name collisions.
CHANGELOG.md Document the workspace merge and e2e sentinel rename.
Cargo.toml Add offchain crates as workspace members; keep solana/ excluded; expand root workspace dependencies to cover offchain crates.
Cargo.lock Large lockfile merge reflecting offchain dependency set in the root workspace.
.github/dependabot.yml Remove offchain workspace scanning entry; keep solana excluded from dependabot cargo updates.
Review details
  • Files reviewed: 48/51 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread offchain/scheduler/native/scheduler_doublezero/Cargo.toml
Base automatically changed from migrate/offchain-path-deps to main September 1, 2026 22:07
@bgm-malbeclabs
bgm-malbeclabs requested a review from a team September 1, 2026 22:07
@bgm-malbeclabs
bgm-malbeclabs force-pushed the migrate/merge-workspaces branch from 090eac9 to aabc99c Compare September 1, 2026 22:09
@bgm-malbeclabs
bgm-malbeclabs enabled auto-merge (squash) September 1, 2026 22:12
@bgm-malbeclabs
bgm-malbeclabs force-pushed the migrate/merge-workspaces branch from 825db3b to cd063da Compare September 1, 2026 22:43
@bgm-malbeclabs
bgm-malbeclabs enabled auto-merge (squash) September 1, 2026 23:01
@bgm-malbeclabs
bgm-malbeclabs force-pushed the migrate/merge-workspaces branch from fd586f7 to 00d7caf Compare September 1, 2026 23:52
Step 4 of the monorepo migration. The offchain tree stops being its own
workspace: `offchain/Cargo.toml`, `offchain/Cargo.lock` and
`offchain/rust-toolchain.toml` are deleted and its 15 crates become root
members, so everything resolves against one lockfile on 1.97.1 and the path
dependencies from #4245 sit inside a single workspace.

`solana/` stays excluded with its own manifest, lockfile and 1.91 toolchain.
62 of the 96 crates in the programs' build closure resolve differently here,
which would change the compiled bytes and break
`solana/programs/sha256sums_*.txt`.

Two inherited keys had to be stated per crate. All 15 crates are edition 2024
and the root workspace is 2021, so an inherited edition would have moved them
to 2021, where they do not build. Four of them also inherited the version, so
an inherited version would have reversioned them from 0.0.1 to 0.38.0.

`bincode` and `reqwest` are declared per crate in the offchain tree instead of
coming from the workspace. This repo is on bincode 2, whose API differs, and on
reqwest with its default TLS backend, while the offchain crates use bincode 1
and rustls with the OS root store. Unifying either is a code change, and the
reqwest one would move `ip_proof`, a trust boundary, onto a different TLS
stack.

The e2e sentinel binary becomes `dz-e2e-sentinel`, because offchain ships a deb
named `doublezero-sentinel` and two members cannot write the same file into
`target/release`. The base image also names the four binaries it serves rather
than building `--workspace`, which would now compile arrow, parquet and the AWS
SDK on the way to a client and a sentinel.

The revdist fixture generator reaches its two solana crates by path rather than
by an unpinned git dependency. Its lockfile had drifted to solana#121 and the
fixtures still regenerate byte-identical.

Offchain's release profile (`lto = true`, `codegen-units = 1`) is not carried
over. Profiles are workspace-wide, so keeping it would apply full link-time
optimization to every release build in this repo.

The spec's plan to convert solana's `>=2,<=3` ranges to exact pins is dropped.
It was written when the programs were joining the workspace. D2 holds them
back and doublezero-shreds still consumes them by git, so the ranges still earn
their keep.
An unscoped `cargo test-sbf` resolves the whole workspace against the
platform-tools rustc, which reports itself as 1.89.0-dev. Folding the offchain
crates in brings `aws-sdk-s3`, which declares a minimum of 1.94.1, and
`rustler`, which declares 1.91, so resolution fails before anything is built:

    error: rustc 1.89.0-dev is not supported by the following packages:
      aws-sdk-s3@1.144.0 requires rustc 1.94.1
      rustler@0.37.4 requires rustc 1.91

Neither crate is in any program's dependency graph. The failure comes from
resolving packages the command never builds.

Passing a `-p` list avoids that resolve and then fails differently: cargo
test-sbf builds the .so for the package in the current directory, and there is
no package here, so solana-program-test finds nothing in target/deploy and
every test panics with "Program file data not available".

Running from each program's directory, the way build-programs already does,
fixes both: the resolve covers one program, and each program's .so is built
before its tests run. Serviceability runs before telemetry, whose test helper
loads the serviceability .so through add_program with no processor.

Verified with the toolchain CI uses, platform-tools v1.54 and rustc 1.89.0:
`make test-sbf` from an emptied target/deploy passes 70 suites with no
failures.
It carried no version before the merge and none after, so cargo called it
0.0.0 either way. Nothing reads it, since the NIF is loaded by the mix release
rather than by cargo, but every other crate in the tree names its own version
and this one may as well.
@bgm-malbeclabs
bgm-malbeclabs force-pushed the migrate/merge-workspaces branch from 00d7caf to 56cb7a0 Compare September 1, 2026 23:52
@bgm-malbeclabs
bgm-malbeclabs merged commit fcbe765 into main Sep 2, 2026
44 of 45 checks passed
@bgm-malbeclabs
bgm-malbeclabs deleted the migrate/merge-workspaces branch September 2, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants