Add storage-subxt crate with static codegen runtime bindings by danielbui12 · Pull Request #282 · paritytech/web3-storage

danielbui12 · 2026-07-13T04:28:05Z

Changes

New crate storage-subxt: generated subxt typed bindings for the runtime, re-exporting subxt and subxt_signer as the single source of truth for those deps across the workspace

Issues

Relevant: [Client] Implement dedicated SDK Checklist #64, Migration: subxt::dynamic → typed storage_subxt bindings #243
Part of: feat: migrate storage-client and provider-node to typed subxt bindings via storage-subxt #239 (splitting changes into smaller PRs)

Introduce crates/storage-subxt, the first crate in the workspace using static subxt codegen (all other subxt consumers are dynamic OnlineClient). It ships generated bindings for the paseo runtime plus committed SCALE metadata, re-exports codec/scale-info/subxt/subxt-core/subxt-signer, and gates runtime selection behind a `mainnet` feature (mainnet artifacts are placeholders until a real mainnet runtime exists). - Workspace: add member, internal path dep, and subxt-core dependency - justfile: add `subxt-codegen` recipe that downloads metadata and regenerates bindings against a running node; metadata and generated code are written to separate files, and the SPDX header is prepended so regeneration keeps files license-compliant - License: crate is Apache-2.0 like sibling library crates, SPDX headers in all sources; generated bindings excluded from hawkeye enforcement

ilchu

Just one doubt about derives being maybe more than needed, but LGTM still.

ilchu · 2026-07-15T09:30:05Z

+    --derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Serialize" \
+    --derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Deserialize" \
+    --derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Serialize" \
+    --derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Deserialize" \
+    --derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Serialize" \
+    --derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Deserialize" \
+    --derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Serialize" \
+    --derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Deserialize" \
+    --derive-for-type "sp_runtime::MultiSignature=codec::Encode" \
+    --derive-for-type "sp_runtime::MultiSignature=codec::Decode" \


I understand all these are good-to-haves, but so far they're not really required by any consumers, right? And as long as we can avoid anything hand-enumerative, I'd rather do that.

I would say yes, we do need. subxt codegen generates the runtime types with only a minimal set of derives.

As my experience in #239, when wiring subxt codegen with rust clients, I faced several errors like the trait X is not implemented for struct....

We could use subxt codegen ...--derive <Derive> to apply to all structs/types . but I prefer
--derive-for-type to scope them to only the types that actually need them.

I am not fan of duplicating the info (this can go out-of-sync easily), can we just reference just subxt-codegen?

I am not fan of duplicating the info (this can go out-of-sync easily), can we just reference just subxt-codegen?

@danielbui12 what about this one?

bkontur · 2026-07-15T10:29:29Z

 ]
 excludes = [
 	"**/*.bin.ts",
+	"crates/storage-subxt/**/*.rs",


why? shouldn't it be apache?

it will be append in generate script

web3-storage/justfile

Lines 255 to 279 in d83397d

subxt-codegen URL=CHAIN_WS OUTPUT="crates/storage-subxt/src/storage_paseo_runtime.rs" METADATA="crates/storage-subxt/metadata/storage_paseo_runtime.scale":

#!/usr/bin/env bash

set -euo pipefail

echo "Downloading metadata from {{ URL }}..."

mkdir -p "$(dirname "{{ METADATA }}")" "$(dirname "{{ OUTPUT }}")"

subxt metadata -f bytes --url "{{ URL }}" > "{{ METADATA }}"

echo "Generating subxt code..."

printf '// SPDX-License-Identifier: Apache-2.0\n\n' > "{{ OUTPUT }}"

subxt codegen --file "{{ METADATA }}" \

--crate "::subxt_core" \

--derive Clone \

--derive Eq \

--derive PartialEq \

--derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Serialize" \

--derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Deserialize" \

--derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Serialize" \

--derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Deserialize" \

--derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Serialize" \

--derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Deserialize" \

--derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Serialize" \

--derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Deserialize" \

--derive-for-type "sp_runtime::MultiSignature=codec::Encode" \

--derive-for-type "sp_runtime::MultiSignature=codec::Decode" \

| rustfmt --edition=2021 --emit=stdout >> "{{ OUTPUT }}"

echo "Generated {{ OUTPUT }}"

bkontur · 2026-07-15T10:32:07Z

+// Mainnet metadata has not been generated yet. Re-exports paseo bindings so
+// --all-features compiles. Replace with real generated bindings once the
+// mainnet metadata file is available (generate like storage_paseo_runtime.rs).
+pub use crate::storage_paseo_runtime::*;


@danielbui12 I would remove mainnet stuff, no TODOs, ok, we could keep mainnet feature

remove storage_mainnet_runtime.rs and all its relevant?

remove storage_mainnet_runtime.rs and all its relevant?

yes, we won't have it for a long time and anyway, these subxt should be generated from the released WASM runtimes, we need some mechanism for comaptibility checking, so we don't need to regenerate and unless not necessary and provider can still work and so on, there is more to it:
#254
#259

I think we could probably merge this, but we will better see with your PR where we want to remove dynamic values to versioned runtime apis...

@bkontur after refactoring the project, I'll rebase #239. that PR migrates dynamic values to versioned runtime apis

bkontur · 2026-07-16T08:42:56Z

@@ -0,0 +1 @@
+// TODO: resolve mainnet scale


@danielbui12 why was this not removed? we discussed to remove all the mainnet stuff..

@bkontur my bad for not checking changes

Regenerates crates/storage-subxt (added on dev in #282 with subxt-codegen 0.44) for subxt 0.50: subxt-core was discontinued at 0.44.3, so the bindings are regenerated with subxt-codegen 0.50.2 against the tracked metadata snapshot (default ::subxt crate path), the crate drops subxt-core and its no_std feature split (subxt 0.50 is std-only), and the justfile codegen recipe loses the --crate ::subxt_core flag.

Brings in the storage-subxt static bindings crate (#282) and converts this branch's new chain access to it as part of the resolution: - chain_events decodes ChallengeCreated / ReplicaAgreementEstablished / ProviderCheckpointSubmitted / BucketCheckpointed through the static event types instead of dynamic name + scale-value field lookups; a shape drift now surfaces as a logged decode failure (backstopped by the safety-net scans) instead of silent None lookups - fetch_challenge point-reads Challenges through the typed static address (unvalidated: bindings are generated from the paseo runtime and the local runtime shares the pallet), replacing the raw-offset decode on this path and picking up the ChunkLocation target shape - pre-existing dynamic call sites (subxt_client scans and tx payloads, chain-state reads, auth membership) are untouched: converting them is a repo-wide follow-up, not part of this branch Also folds the event fan-out tests into the coordinators test binary, reusing its shared helpers (test_state, wait_for, and the committed- chunk fixture now hosted in main.rs) instead of duplicating them in a separate integration target.

danielbui12 requested review from bkontur and ilchu and removed request for bkontur July 13, 2026 04:28

danielbui12 self-assigned this Jul 13, 2026

danielbui12 mentioned this pull request Jul 13, 2026

Pin pnpm to 11.11.0 via packageManager field #283

Merged

Merge branch 'dev' into storage-subxt

325edb7

danielbui12 enabled auto-merge July 14, 2026 02:51

danielbui12 added 2 commits July 14, 2026 10:23

Merge branch 'dev' into storage-subxt

73b4e1d

Merge branch 'dev' into storage-subxt

4c06c6e

ilchu approved these changes Jul 15, 2026

View reviewed changes

chore: ignore storage-subxt in rust cov check

d83397d

danielbui12 added this pull request to the merge queue Jul 15, 2026

bkontur removed this pull request from the merge queue due to a manual request Jul 15, 2026

bkontur reviewed Jul 15, 2026

View reviewed changes

Comment thread crates/storage-subxt/src/lib.rs Outdated

bkontur reviewed Jul 15, 2026

View reviewed changes

danielbui12 added 2 commits July 15, 2026 17:52

chore: remove unused mainnet subxt codegen

195783e

Merge branch 'dev' into storage-subxt

5ed1d46

danielbui12 added this pull request to the merge queue Jul 16, 2026

Merged via the queue into dev with commit 3ffda77 Jul 16, 2026
38 checks passed

danielbui12 deleted the storage-subxt branch July 16, 2026 03:00

bkontur reviewed Jul 16, 2026

View reviewed changes

	subxt-codegen URL=CHAIN_WS OUTPUT="crates/storage-subxt/src/storage_paseo_runtime.rs" METADATA="crates/storage-subxt/metadata/storage_paseo_runtime.scale":
	#!/usr/bin/env bash
	set -euo pipefail
	echo "Downloading metadata from {{ URL }}..."
	mkdir -p "$(dirname "{{ METADATA }}")" "$(dirname "{{ OUTPUT }}")"
	subxt metadata -f bytes --url "{{ URL }}" > "{{ METADATA }}"
	echo "Generating subxt code..."
	printf '// SPDX-License-Identifier: Apache-2.0\n\n' > "{{ OUTPUT }}"
	subxt codegen --file "{{ METADATA }}" \
	--crate "::subxt_core" \
	--derive Clone \
	--derive Eq \
	--derive PartialEq \
	--derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Serialize" \
	--derive-for-type "pallet_storage_provider::pallet::ProviderInfo=serde::Deserialize" \
	--derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Serialize" \
	--derive-for-type "pallet_storage_provider::pallet::ProviderSettings=serde::Deserialize" \
	--derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Serialize" \
	--derive-for-type "pallet_storage_provider::pallet::ProviderStats=serde::Deserialize" \
	--derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Serialize" \
	--derive-for-type "bounded_collections::bounded_vec::BoundedVec=serde::Deserialize" \
	--derive-for-type "sp_runtime::MultiSignature=codec::Encode" \
	--derive-for-type "sp_runtime::MultiSignature=codec::Decode" \
	\| rustfmt --edition=2021 --emit=stdout >> "{{ OUTPUT }}"
	echo "Generated {{ OUTPUT }}"

		@@ -0,0 +1 @@
		// TODO: resolve mainnet scale No newline at end of file

Uh oh!

Conversation

danielbui12 commented Jul 13, 2026

Changes

Issues

Uh oh!

ilchu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielbui12 Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danielbui12 Jul 16, 2026 •

edited

Loading