Use `scale-typegen` as a backend for the codegen #1260

tadeohepperle · 2023-11-13T17:15:00Z

This PR uses scale-typegen to generate the runtime_types module and structs for calls, events, etc.
Let's see if the pipeline works, I manually checked the differences in the polkadot.rs file to make sure the differences are valid (almost none).

Recursive Derives

PR also includes adjustments to the subxt macro to allow for recursive derives. Shown in the setup_config_custom.rs example.

Fixes #1268
Fixes #1247 (indirectly)

Type Aliases

I tried to reconcile the conflicts between the master and this branch, and implemented the alias modules a bit differently, I hope it does not break anything.

…type-generation

* lightclient: Close wasm socket while dropping from connecting state Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Construct one time only closures Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Enable console logs for lightclient WASM testing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Separate wakes and check connectivity on poll_read Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket depending on internal state Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "lightclient: Separate wakes and check connectivity on poll_read" This reverts commit 8660940. * lightclient: Return pending if socket is opening from poll_read Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket on `poll_close` Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reset closures on Drop to avoid recursive invokation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket if not already closing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli: Add chainSpec command Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli/chainSpec: Move to dedicated module Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Compute the state root hash Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Remove code substitutes Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Update polkadot.json Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * scripts: Generate the chain spec Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Remove testing artifacts Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Fix clippy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Apply rustfmt Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Introduce feature flag for smoldot dependency Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Rename chain-spec to chain-spec-pruning Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * scripts: Update chain-spec command Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

cli/src/commands/codegen.rs

codegen/src/api/mod.rs

codegen/src/api/runtime_apis.rs

subxt/examples/setup_config_custom.rs

* update crypto dependencies, adjust keypair * add scale_info::TypeInfo derive in some places * add multi signature derive

lexnv · 2024-01-09T10:43:55Z

testing/integration-tests/src/full_client/codegen/polkadot.rs

It seems that the changes have not affected the aliases and other code generations (except for additional docs, which should be fine). Just to double check, is this file generated from the latest polkadot with all the code changes included right? 🤔

Yes, I ran the polkadot.rs file generation command after implementing all the code changes in this PR. :)

lexnv · 2024-01-09T10:46:55Z

codegen/src/lib.rs

@@ -167,8 +167,16 @@ impl CodegenBuilder {
        &mut self,


We'd need to adjust the docs of this function a bit, the warning section specifically

lexnv · 2024-01-09T10:47:24Z

codegen/src/lib.rs

@@ -181,11 +189,19 @@ impl CodegenBuilder {
        &mut self,


Same doc adjustment here, since we now have the recursive parameter

lexnv · 2024-01-09T10:52:40Z

cli/src/commands/codegen.rs

@@ -168,7 +168,8 @@ fn codegen(
            .map_err(|e| eyre!("Cannot parse derive for type {ty_str}: {e}"))?;
        let derive = syn::parse_str(&derive)
            .map_err(|e| eyre!("Cannot parse derive for type {ty_str}: {e}"))?;
-        codegen.add_derives_for_type(ty, std::iter::once(derive));
+        // Note: recursive derives and attributes not supported in the CLI => recursive: false


Would we want to support recursive derives in the CLI? Might be worth a separate issue to remind us of it

Hmm, maybe. Or maybe we just make everything recursive by default in the CLI, I think it is already quite hard to specify derives in the CLI.

Would be great if you could open an issue for it :)

codegen/src/api/calls.rs

lexnv · 2024-01-09T10:57:18Z

codegen/src/api/calls.rs

                    .iter()
                    .map(|(name, field)| {
-                        let call_arg = if field.is_boxed() {
+                        // Note: fn_arg_type this is relative the type path of the type alias when prefixed with `types::`, e.g. `set_max_code_size::New`


Could there be a variant of the metadata where this does not correspond to the expected alias? Or it would be different from the previous format_ident!("{}", name.to_string().to_upper_camel_case())?

Good question, when calling generate_structs_from_variants, which calls generate_type_alias_mod for each such composite, for each field, the field.type_path is replaced by #alias_mod_name::#alias_name, where alias_name is either format_ident!("{}", name.to_string().to_upper_camel_case()) (named fields) or format_ident!("Field{}", i) for unnamed fields, so the logic should be very similar to what you implemented before. I cannot think of a case right now where this is not fulfilled.

codegen/src/api/constants.rs

lexnv · 2024-01-09T11:03:01Z

codegen/src/api/errors.rs

-    let error_type = type_gen.resolve_type_path(error_ty);
-    let error_ty = type_gen.resolve_type(error_ty);
+    let error_type = type_gen.resolve_type_path(error_ty)?;
+    let error_ty = type_gen.resolve_type(error_ty)?;
    let docs = &error_ty.docs;
    let docs = should_gen_docs


Could this be infered from the type_gen.settings()?

Oh, totally! Good catch!

lexnv · 2024-01-09T11:08:34Z

codegen/src/api/mod.rs

-        derives: DerivesRegistry,
-        type_substitutes: TypeSubstitutes,
+        derives: scale_typegen::DerivesRegistry,
+        type_substitutes: scale_typegen::TypeSubstitutes,
        crate_path: syn::Path,
        should_gen_docs: bool,


Is there any particular reason for having this? It is only passed to the error generation, while others use the type_gen settings

What do you mean? Having scale_typegen::DerivesRegistry specified fully, is not necassary, I should just import DerivesRegistry. Maybe I understand the question wrong.

Ops, I forgot to type should_gen_docs. I do like having the scale_typegen:: prefix 👍

Ah okay, the should_gen_docs we actually need because the typegen settings are not built up yet. They are only built in this very function using the crate_path, should_gen_docs, substitutes and derives :)

lexnv · 2024-01-09T11:20:13Z

The code looks clean and simplifies the code quite a lot! Nice one! 👍

Just some nits about the documentation and inferring the type alias paths

…type-generation

niklasad1 · 2024-01-10T10:59:43Z

codegen/src/api/mod.rs

@@ -12,20 +12,20 @@ mod events;
 mod runtime_apis;
 mod storage;

+use scale_typegen::typegen::ir::type_ir::{CompositeFieldIR, CompositeIR, CompositeIRKind};


would be nice to export all these types from the root in scale_typegen all the types are already unique and I don't think for instance scale_typegen::typegen::ir::type_ir::CompositeFieldIR is necessary, thus scale_typegen::CompositeFieldIR or ``scale_typegen::ir::CompositeFieldIR` would be sufficient.

Nothing to address here just a comment

Good point! We already expose some other types at the root, I did not anticipate that we would need easy access to these structs.

codegen/src/api/custom_values.rs

codegen/src/api/mod.rs

codegen/src/api/storage.rs

codegen/src/api/mod.rs

niklasad1

Looks great to me

niklasad1 · 2024-01-11T09:21:15Z

cli/src/commands/diff.rs

@@ -309,12 +309,12 @@ fn storage_differences<'a>(
        |e| {
            pallet_metadata_1
                .storage_hash(e.name())
-                .expect("storage entry should be present")
+                .expect("storage entry is in medadata; qed")


Suggested change

.expect("storage entry is in medadata; qed")

.expect("storage entry is in metadata; qed")

niklasad1 · 2024-01-11T09:21:26Z

cli/src/commands/diff.rs

        },
        |e| {
            pallet_metadata_2
                .storage_hash(e.name())
-                .expect("storage entry should be present")
+                .expect("storage entry is in medadata; qed")


Suggested change

.expect("storage entry is in medadata; qed")

.expect("storage entry is in metadata; qed")

lexnv

Nice one! 👍

tadeohepperle added 5 commits November 9, 2023 18:21

integrate scale-typegen, remove types mod

e163e57

reintroduce default substitutes and derives

1760792

support runtime_types only again

b297fc8

generating polkadot.rs ok

7a2678a

Merge branch 'master' into tadeohepperle/implement-scale-typegen-for-…

60c1e62

…type-generation

tadeohepperle requested a review from a team as a code owner November 13, 2023 17:15

update scale-typegen to discrete error types

297d2c6

tadeohepperle mentioned this pull request Nov 15, 2023

Handling skipped type params in subxt-codegen #1247

Closed

tadeohepperle added 3 commits November 21, 2023 14:47

scale-typegen-api-changes

f47fd4e

add note about UncheckedExtrinsic in default substitutes

03591a6

Merge branch 'master' into tadeohepperle/implement-scale-typegen-for-…

9e56546

…type-generation

tadeohepperle force-pushed the tadeohepperle/implement-scale-typegen-for-type-generation branch from 7db70a1 to 03591a6 Compare November 21, 2023 14:01

tadeohepperle and others added 8 commits November 28, 2023 18:41

add resursive attributes and derives

50ed54e

adjust example where Clone bound recursive

dabc88c

Merge branch 'master' into tadeohepperle/implement-scale-typegen-for-…

cd73d31

…type-generation

move scale-typegen dependency to workspace

fd718bf

expose default typegen settings

985a46c

workflows: Install rustup component for building substrate (#1295)

44c42c3

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

tadeohepperle requested a review from a team as a code owner November 29, 2023 13:02

conflicts

ee1e096

niklasad1 reviewed Nov 30, 2023

View reviewed changes

cli/src/commands/codegen.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Nov 30, 2023

View reviewed changes

codegen/src/api/mod.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Nov 30, 2023

View reviewed changes

codegen/src/api/mod.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Nov 30, 2023

View reviewed changes

codegen/src/api/runtime_apis.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Nov 30, 2023

View reviewed changes

subxt/examples/setup_config_custom.rs Show resolved Hide resolved

tadeohepperle and others added 3 commits November 30, 2023 16:56

remove comments and unused args

58b8fee

Update substrate- and signer-related dependencies (#1297)

0c7d373

* update crypto dependencies, adjust keypair * add scale_info::TypeInfo derive in some places * add multi signature derive

fix lock file

42643d8

lexnv reviewed Jan 9, 2024

View reviewed changes

codegen/src/api/calls.rs Show resolved Hide resolved

lexnv reviewed Jan 9, 2024

View reviewed changes

codegen/src/api/constants.rs Show resolved Hide resolved

lexnv reviewed Jan 9, 2024

View reviewed changes

tadeohepperle added 4 commits January 9, 2024 14:53

implemented Alex suggestions, regenerated polkadot.rs (did not change)

3d1fdec

Merge branch 'master' into tadeohepperle/implement-scale-typegen-for-…

f0ef2a9

…type-generation

Merge branch 'master' into tadeohepperle/implement-scale-typegen-for-…

72fee7d

…type-generation

resolve conflicts in Cargo.lock

a96c90f