Generate runtime API from metadata #294

ascjones · 2021-09-15T12:35:57Z

Implements macros which given the SCALE encoded metadata from a substrate runtime, will generate an api similar to polkadot.js

Closes #97, and probably many more issues...

todo

Generated type derives
- Consider adding Clone derive to all generated types.
- Configure macro to generate additional default derives for generated types.
- Specify derives to add to specific generated types?
Make tests all pass
Resolve all todos in code
more todos...

dvdplm · 2021-10-29T19:06:26Z

codegen/src/api/storage.rs

    let fn_name = format_ident!("{}", storage_entry.name.to_snake_case());
+    let fn_name_iter = format_ident!("{}_iter", fn_name);


Is it possible to break things with bad storage entry naming, e.g. a storage entry named account_iter and another named account (which would gain an account_iter method)? A very contrived example, I know. I guess the real question is: do we need to check Idents for clashes?

Indeed it is entirely possible to break this. In the end the compiler will check for clashes for us so I don't see it as a high priority to solve at this point.

ascjones · 2021-11-03T11:29:25Z

Please see #309 for follow-ups to this PR

* Remove test macro * Remove client crate * Create tests crate and move pallet specific tests there * Extract client, remove metadata and extra, more demolition * Update substrate dependencies to git dependencies * Remove Store stuff for now * Comment out some Call usages * Add back Runtime trait coped from original System trait * Make subxt lib compile * Delete old proc macros and copy over type generation from chameleon * WIP make transfer balance test pass * Change to subxt attribute macro * WIP provide user defined type substitutes * User defined type substitutes compile * WIP submitting transactions * WIP transfer balance test * Fix macro * Cargo fmt * WIP generating storage hashers * WIP add AccountData trait for fetching the nonce * Support single type storage map keys * WIP impl AccountInfo retrieval * Fix up storage struct generation * Implement AccountData triait directly on storage entry * Borrow storage map key and convert account id * Implement storage fetch client methods * Remove legacy metadata storage key construction * Rename CheckEra to CheckMortality * Substitute perthings types for compact impls * Fmt * Downgrade dyn-clone for cargo-contract compat * Scale-fo 1.0 * scale-info 1.0 * Remove special range handling * Restore wildcard type params * Frame metadata 14.0 * WIP decoding events * WIP more dynamically decoding events * Fmt * Decode events, handle errors * Uncomment some tests * Remove unused get_mod function * Fix some warnings * Fix some more warnings * Fix some more warnings * Add tests mod * Rename node-runtime tests mod to frame * Fix some warnings * Fmt * WIP generate storage client with getters * Storage client compiling * Generate storage client api * Fix up system account query account ids * WIP generating tx api fns * Only generate tx api fields when calls available * Fix tx api call fns * Fmt * WIP generate event structs * call functions not async * Derive Eq for comparison on generated types * Generate event structs * Fix call name * Fmt * Update node runtime metadata to substrate c000780db * Download latest substrate release for integration testing * Fix event decoding * Remove unused imports * Fix plain storage access, total_issuance pass * Fmt * Restore contracts tests * Backoff connecting to substrate node * Add required TypeInfo impls for local SignedExtension impls * Remove unnecessary assert formatting * Fix handling of DispatchError * Refactor contracts tests * Troubleshooting contract not found * Remove more client feature stuff * Fix dynamic event variant decoding, write consumed index to output * Fmt * Use substrate branch with heavy dependency removed * Remove sp-rcp dependency, define types locally * Ignore cargo timeing files * Use my branch for substrate test deps * Fix storage key type gen * Comment out fetching contract info * Add key iteration, extract storage client from main client * Debugging key generation * Use substrate master branch * Fix call test * Remove TypeSegmenter and dynclone dependency * Publicly expose Rpc mod * Unused import warnings * Add getter for runtime metadata * Add pallet and event indices for raw events * Add is_call and is_event convenience trait functions * Add missing docs * Refactor tests crate * Restore remaining client tests * Fmt * Fix warnings * Restore get_mod as test helper and fmt * Use client references for api calls * Fix api usages with methods * Use Bytes for RawEvent debug * Update metadata * Restoring some Balances tests * Populate runtime storage metadata * Restore balances lock test * Restore Balances error test * Fmt * Restore transfer subscription API * Staking test * Restore another staking test * Restore another staking test * Restore another staking test * Partially restore chill_works_for_controller_only staking test * Fix fetching Optional storage entries * Restore staking bond test * Restore remaining staking tests * Fmt * Restore sudo tests * Add some system tests * Fmt * Resolve some todos * Remove pass through rpc methods on Client, expose via rpc() getter * Remove more rpc pass through methods * Remove submit tx pass through rpc methods * Add some comments to SubmittableExtrinsic methods * Construct the runtime api from the client * Fmt * Use From trait instead of new for AccountData query * Rename subxt_proc_macro crate to subxt_macro * Fix AccountData From impl * Extract codegen crate from macro crate * Fmt * Replace chameleon hidden field name * Extract StructDef for generating structs * More refactoring of StructDef, moving towards sharing with typegen * Replace explicit tests crate with single implicit integration tests crate * Rename from substrate-subxt to subxt * Fix runtime path relative to root Cargo.toml * Move RpcClient creation to RpcClient * WIP get examples to compile * Rename Runtime to Config trait * WIP implementing default Config * WIP implementing default extrinsic extras * fix metadata constants (paritytech#299) * Move DefaultConfig definition and impl to macro * Extract type substitute parsing to ir mod * Extract calls, events and storage from api generation * Add some hardcoded type substitute defaults * Fmt * Add utility pallet tests (paritytech#300) * add batch call test example * add pallet utility tests * add utility module * fix warnings * Add polkadot runtime metadata for example * Fix system errors and fmt * Add subxt-cli crate * Add metadata and codegen subcommands * Make subxt-cli codegen command work * Fmt * Add polkadot codegen test * Comment about how to run codegen * Derive AsCompact for structs with single concrete unsigned int field * Fix bitvec codegen, adds as non optional dependency * Regenerate polkadot api with bitvec fix * Edition 2021 * Fix polkadot codegen with bitvec * Polkadot balance transfer is working * Fix fetch remote * Fix transfer_subscribe example * Fix submit_and_watch example * Fmt * Generate storage iter method for iterating over keys * Fmt * Fix existential deposit test * Fix staking tests * Add option for custom generated type derives * Add generated type derives for test runtime api * Fmt * Copy WrapperTypeOpaque from substrate, add Encode/Decode * Fmt * Extract type generator to module, separate & fix tests * Fully qualified primitive and prelude types * Fix up remaining type gen tests * Skip formatting of generated polkadot example code * Remove empty utility test file. * Newline * Update cli/src/main.rs Co-authored-by: David <dvdplm@gmail.com> * Rename subxt-cli executable to subxt * Update src/client.rs Co-authored-by: David <dvdplm@gmail.com> * Add some code docs to TypeGenerator. * Extract TypePath to own file * Extract type def generation to separate file * Renamed ModuleType to TypeDefGen * Fmt * Factor out type parameter from final_key * Fix some type paths * Resolve some todos * Resolve some panic todos in events * Add EventsDecodingError * Decode compact composite types with a single primitive field * Decode compact composite types with a single primitive field * Update src/metadata.rs Co-authored-by: Andrew Plaza <aplaza@liquidthink.net> * Remove Perbill compact substitute types * Remove todos regarding maintaining Rust code items, promoted to follow up issue. * Remove todo regarding overridding default config impl * Remove todo regarding overridding default Extra * Remove todo regarding AccountData storage type defintion * Remove todo regarding borrowing storage key arguments * Remove type substitution tests todo * Remove `Box` field name type hack todo * Remove Compact todo * Remove sudo todos * Remove BitVec implementation todo * Fmt * Add health warning to README * Fix up health warning Co-authored-by: Paulo Martins <paulormart@users.noreply.github.com> Co-authored-by: David <dvdplm@gmail.com> Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

ascjones added 30 commits August 26, 2021 12:21

Remove test macro

f43df0e

Remove client crate

01b7131

Create tests crate and move pallet specific tests there

ff6b9fc

Extract client, remove metadata and extra, more demolition

d7c1d22

Update substrate dependencies to git dependencies

b755c29

Remove Store stuff for now

05fe051

Comment out some Call usages

5e91406

Add back Runtime trait coped from original System trait

59aa4d7

Make subxt lib compile

e337d34

Delete old proc macros and copy over type generation from chameleon

8540533

WIP make transfer balance test pass

e917dac

Change to subxt attribute macro

2ef017d

WIP provide user defined type substitutes

f3005cb

User defined type substitutes compile

e4f1374

WIP submitting transactions

b98b7aa

WIP transfer balance test

e20f123

Fix macro

2cf09c1

Cargo fmt

56cb2a6

WIP generating storage hashers

c35d781

WIP add AccountData trait for fetching the nonce

4726db3

Support single type storage map keys

f8f8b47

WIP impl AccountInfo retrieval

4b573f5

Fix up storage struct generation

a6e7225

Implement AccountData triait directly on storage entry

d2bd669

Borrow storage map key and convert account id

11248d0

Implement storage fetch client methods

b9ad905

Remove legacy metadata storage key construction

f98d95a

Rename CheckEra to CheckMortality

a8ca64a

Substitute perthings types for compact impls

9eac486

Fmt

a5281fc

ascjones added 3 commits November 3, 2021 10:12

Remove Compact todo

ba2e3a8

Remove sudo todos

06a6ddf

Remove BitVec implementation todo

0d07b0c

ascjones mentioned this pull request Nov 3, 2021

Follow ups to https://github.com/paritytech/subxt/pull/294 #309

Closed

18 tasks

Fmt

f0bb889

dvdplm approved these changes Nov 3, 2021

View reviewed changes

ascjones added 2 commits November 3, 2021 11:06

Add health warning to README

2c93f3c

Fix up health warning

641ac27

ascjones merged commit 793c945 into master Nov 3, 2021

This was referenced Nov 3, 2021

Add batch calls example #303

Closed

Codegen: enum variant indices #308

Merged

Reexport support crates #260

Closed

h4x3rotab mentioned this pull request Nov 5, 2021

Workaround for pallet missing declare error type #259

Closed

ascjones mentioned this pull request Nov 25, 2021

Configurable extrinsic Extra #331

Closed

jsdw mentioned this pull request Feb 1, 2022

Address niggles from #294 #419

Closed

6 tasks

ascjones deleted the aj-metadata-vnext branch April 1, 2022 08:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate runtime API from metadata #294

Generate runtime API from metadata #294

ascjones commented Sep 15, 2021 •

edited

Loading

dvdplm Oct 29, 2021

ascjones Nov 3, 2021

ascjones commented Nov 3, 2021

		let fn_name = format_ident!("{}", storage_entry.name.to_snake_case());
		let fn_name_iter = format_ident!("{}_iter", fn_name);

Generate runtime API from metadata #294

Generate runtime API from metadata #294

Conversation

ascjones commented Sep 15, 2021 • edited Loading

todo

dvdplm Oct 29, 2021

Choose a reason for hiding this comment

ascjones Nov 3, 2021

Choose a reason for hiding this comment

ascjones commented Nov 3, 2021

ascjones commented Sep 15, 2021 •

edited

Loading