CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration #1290

tadeohepperle · 2023-11-24T08:21:27Z

Warning: Do not merge (we need to wait for a full scale-typgen release before we can integrate this PR)

This PR restructures the subxt explore CLI command. You can now explore pallet events: subxt explore pallet balances events:

And you can explore runtime apis: subxt explore api metadata by adding the --execute (-e) flag to a runtime api method, e.g. subxt explore api core version -e you can view the return-value of the runtime-api call:

Upon starting the explore command (subxt explore) you get shown a list of all available pallets and runtime apis. When You only type subxt explore pallet you only see available pallets. For outputs I use ANSI coloring which is already part of the color_eyre crate, so no new dependencies needed to be added. I added a dependency though for indoc, which makes the code a lot cleaner.

Testing

Manually tested, and tests have been adjusted.

* config: Add `SkipCheckIfFeeless` signed extension Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Add extra extension to the default params Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * examples: Adjust signed extension example Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Extend SkipCheckIfFeeless with inner signed extension Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Configure SkipCheck with inner signed extension params Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Implement Deafult for SkipCheckIfFeelessParams with Option Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * examples: Fix example with proper extension Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Extend <T as Config>::AssetId with EncodeAsType and Clone Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Add SkipCheck with AssetTx Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Encode as type from metadata the inner signed extensions Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Adjust examples Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * blocks: Use `SkipCheckIfFeeless` for decoding the tip of extensions Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Decode `SkipCheckIfFeeless` with `Self` Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * tests: Adjust testing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Descriptive errors for building `SkipCheckIfFeeless` Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Add docs for extra error types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Add extra derives to signed extensions Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * config: Use `Default::default` to simplify type init Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Make `smoldot::chainID` part of the RPC requests Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Make `BackgroundTask` generic over `PlatformRef` and chain Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Construct from raw smoldot and target different chains Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Update cargo lock for wasm tests Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reuse `new_from_client` method and removed unused imports Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reexport smoldot client and RPC objects used in pub interface Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Adjust `new_from_client` interface Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Extend background to poll over multiple RPC objects Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Build light client from raw and target different chains Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Add demo chain specs Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Move artifacts to dedicated folder Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Use SelectAll to drive all streams Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Fetch initial data from the target chain Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reexport other smoldot objects Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Target chain with potentially different config Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt/rpc: Log chainID for debugging Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt/examples: Add smoldot client with parachain example Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Propagate chain ID together with rpc responses object Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Multiplex responses by request ID and chain ID Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Add raw light client builder Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Add cargo feature flag for parachains example Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Derive default for internal structure Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Guard reexports by std feature flag Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update subxt/src/client/light_client/mod.rs Co-authored-by: James Wilson <james@jsdw.me> * lightclient: Update the builder pattern and chain targetting Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Fix documentation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Provide more insightful docs wrt native/wasm panics Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * examples: Adjust comment location Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Refactor UniqueChainId into the background task Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update lightclient/src/background.rs Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com> * Update subxt/src/client/light_client/builder.rs Co-authored-by: James Wilson <james@jsdw.me> * lightclient: Update docs wrt panics Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Update docs wrt to smoldot instance -> client Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Use IntoIter instead of Iterator Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Adjsut docs wrt [`Self::new_from_client`] Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Remove RawRpc from LightClient in favor of chainID Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reexport everything under smoldot module Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Use stateRootHash instead of genesis.raw Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.28 to 0.3.29. - [Release notes](https://github.com/rust-lang/futures-rs/releases) - [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md) - [Commits](rust-lang/futures-rs@0.3.28...0.3.29) --- updated-dependencies: - dependency-name: futures dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [zeroize](https://github.com/RustCrypto/utils) from 1.6.0 to 1.7.0. - [Commits](https://github.com/RustCrypto/utils/commits) --- updated-dependencies: - dependency-name: zeroize dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.17 to 0.3.18. - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](tokio-rs/tracing@tracing-subscriber-0.3.17...tracing-subscriber-0.3.18) --- updated-dependencies: - dependency-name: tracing-subscriber dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.10 to 0.2.11. - [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md) - [Commits](rust-random/getrandom@v0.2.10...v0.2.11) --- updated-dependencies: - dependency-name: getrandom dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* impl RpcClientT for Arc<WsClient> * fix grumbles: impl for Box<T> and Arc<T> * grumbles: move RpcClientT impls

* improve `substrate-compat` * From => Into --------- Co-authored-by: James Wilson <james@jsdw.me>

Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.69 to 1.0.70. - [Release notes](https://github.com/dtolnay/proc-macro2/releases) - [Commits](dtolnay/proc-macro2@1.0.69...1.0.70) --- updated-dependencies: - dependency-name: proc-macro2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.192 to 1.0.193. - [Release notes](https://github.com/serde-rs/serde/releases) - [Commits](serde-rs/serde@v1.0.192...v1.0.193) --- updated-dependencies: - dependency-name: serde dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* lightclient: Close wasm socket while dropping from connecting state Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Construct one time only closures Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Enable console logs for lightclient WASM testing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Separate wakes and check connectivity on poll_read Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket depending on internal state Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "lightclient: Separate wakes and check connectivity on poll_read" This reverts commit 8660940. * lightclient: Return pending if socket is opening from poll_read Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket on `poll_close` Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Reset closures on Drop to avoid recursive invokation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Close the socket if not already closing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpcmethods * followstr * mod and else

…1352) * github CI action cronjob * add commit message * fix the CI yml files * binary crate for CI script with substrate-runner * update the CI script * correct the artifacts script * remove bash script

…old (#1372) * testing(fix): Ensure lightclient chainSpec is at least one block old Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "testing(fix): Ensure lightclient chainSpec is at least one block old" This reverts commit 0eafcb2. * lightclient(fix): Ensure lightclient chainSpec is at least one block old Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Link smoldot issue Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Use tokio under lightclient feature flag Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Do not sleep on errors to fetch the chainSpec Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Remove test file Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * lightclient: Subscribe to two finalized blocks Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Revert cargo toml Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Reduce the light client timpeut to 15 seconds Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * ci: Use ubuntu-latest for light-client tests Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* asset hub example and book adjustment * formatting * recursive derives * polkadot monitor example and book adjustments * formatting * adjust docs and examples, add dynamic example * james suggestions * fmt * chore(subxt/src): typo fix (#1370) * rpcmethods * followstr * mod and else * Weekly Cronjob fetching artifacts and generating polkadot.rs file. (#1352) * github CI action cronjob * add commit message * fix the CI yml files * binary crate for CI script with substrate-runner * update the CI script * correct the artifacts script * remove bash script --------- Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: Pan chao <152830401+Pan-chao@users.noreply.github.com>

tadeohepperle · 2024-01-18T16:21:51Z

I finally figured out how to submit the parameters needed for a dynamic runtime api call in the CLI:

take the input as a &str
parse it into a scale value
encode it to bytes using the params type id
decode it from the bytes using the params type id

IIRC @niklasad1 was doing something similar a few months back somewhere.

Now we can execute runtime apis like this:

lexnv · 2024-01-19T08:48:22Z

cli/Cargo.toml

@@ -40,5 +40,12 @@ scale-value = { workspace = true }
 syn = { workspace = true }
 jsonrpsee = { workspace = true, features = ["async-client", "client-ws-transport-native-tls", "http-client"] }
 tokio = { workspace = true, features = ["rt-multi-thread"] }
+scale-typegen-description = { workspace = true }
+heck = { workspace = true }
+indoc = "2"


We could probably move the indoc adn strip-ansi to the workspace Cargo.toml to make sure we can easily update it everywhere. Just in case we'll add it to multipl crates in the future. Leaving this up to you

Good idea, but I would prefer to do it later when we really need it another crate of the workspace.

Let's do it now for consistency, because I think every other dependency is now listed in the workspace thing first (except for non-workspace crates where we can't). Makes it mostly easy to see the entire set of crates we use in one place at least!

jsdw · 2024-01-19T14:19:23Z

cli/src/commands/explore/pallets/events.rs

+
+    let usage = || {
+        let events = events_to_string(event_variants, pallet_name);
+        formatdoc! {"


This does make it easier to see how things will be formatted; good find!

jsdw · 2024-01-19T14:27:08Z

cli/src/commands/explore/pallets/storage.rs

+    trailing_args: Vec<String>,
+}
+
+pub async fn explore_storage(


With runtime APIs, we have an -e flag to actually try to execute a runtime API call. I feel like we should do the same for executing a storage request, so that by default we don't try to make a request but we can if we want. What do you think?

Okay, I added an -e flag for storage entries as well and made the API very similar to the runtime apis.

jsdw · 2024-01-19T14:29:55Z

cli/src/commands/explore/runtime_apis/mod.rs

+    #[allow(non_snake_case)]
+    let INPUT_VALUE: String = "<INPUT_VALUE>".blue().to_string();


Personally I'd just name it something like blue_input_placeholder or something (likewise elsewhere) and avoid needing to add the allow

jsdw · 2024-01-19T14:31:17Z

cli/src/commands/explore/runtime_apis/mod.rs

+        {fields_example}"}
+    };
+
+    let execute_usage = || {


This could just be a block?

Oh, good catch, I made it a closure to use it twice below, but used it only once. I am now replacing the usage() below by the second execute_usage() as intended.

jsdw · 2024-01-19T14:33:27Z

cli/src/commands/explore/mod.rs

 /// # Encoded call data:
 /// # 0x24040607020cffffff00
 /// ```
-/// ## Constants
+/// ### Constants


I'd ensure a newline above and below each heading (this and below are missing one or both), and in general add newlines before and after the code blocks in this whole section (and anywhere else) just to make it more readable as plain text :)

jsdw · 2024-01-19T14:37:51Z

Minor comments but I think they should be addressed, and then for me this is good to go! The colouring thing on my laptop is strange but we can sort that out later if it keeps coming up.

jsdw

Awesome stuff, nicely done :)

tadeohepperle and others added 22 commits November 20, 2023 10:12

restructure cli commands

38f3a16

subxt: Replace removed lint (#1270)

6980c2f

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

impl RpcClientT for Arc<T> and Box<T> (#1277)

871561c

* impl RpcClientT for Arc<WsClient> * fix grumbles: impl for Box<T> and Arc<T> * grumbles: move RpcClientT impls

first iteration of using scale_typegen

9f74bca

Merge branch 'master' into tadeohepperle/cli-support-runtime-apis

bbe8e4f

introduce indoc for formatting

5bf6c52

calls, constants and home are cleaner now

8052211

added event subcommand

5fd50ce

show runtime apis working

c5973a0

add better code formatting

ec8ac18

fix style

f6b8ddd

adjust tests, use owo_colorize to not add extra dependency

af019a6

fmt

b871375

adjust docs

52b4013

Merge branch 'master' into tadeohepperle/cli-support-runtime-apis

febd3a0

tadeohepperle requested a review from a team as a code owner November 24, 2023 08:21

tadeohepperle changed the title ~~CLI Tool Explore Command: Support Runtime APIs, Events, Colorized outputs, Scale-typegen integration~~ CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration Nov 24, 2023

tadeohepperle and others added 6 commits November 29, 2023 10:52

move scale-typegen-description dependency to workspace

cd3f660

improve substrate-compat (#1265)

a7835f6

* improve `substrate-compat` * From => Into --------- Co-authored-by: James Wilson <james@jsdw.me>

workflows: Install rustup component for building substrate (#1295)

9040872

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

jsdw and others added 13 commits January 18, 2024 17:07

megre in light client test change pr

d3777fa

chore(subxt/src): typo fix (#1370)

d95c8eb

* rpcmethods * followstr * mod and else

Weekly Cronjob fetching artifacts and generating polkadot.rs file. (#…

e39acda

…1352) * github CI action cronjob * add commit message * fix the CI yml files * binary crate for CI script with substrate-runner * update the CI script * correct the artifacts script * remove bash script

actually only wait for machete+fmt, clippy can be much slower

26607f1

update CI file from Alex PR

fce3ef7

resolve clippy err

240cf26

Try a few RPC nodes in case one of them is not working

1c37385

fix submit_transaction debug logging of message

833beb2

fix formatting of returned sections

9e9dcfa

Merge branch 'master' into tadeohepperle/cli-support-runtime-apis

d9570df

niklasad1 approved these changes Jan 18, 2024

View reviewed changes

Merge branch 'master' into tadeohepperle/cli-support-runtime-apis

e12dabc

lexnv reviewed Jan 19, 2024

View reviewed changes

jsdw reviewed Jan 19, 2024

View reviewed changes

jsdw mentioned this pull request Jan 19, 2024

CLI tool: recursive derives and attributes #1379

Merged

make storage use execute flag as well

e1483f3

tadeohepperle mentioned this pull request Jan 19, 2024

CLI tool: ANSI colors cause output to be greyed out on Mac #1382

Open

jsdw approved these changes Jan 19, 2024

View reviewed changes

tadeohepperle merged commit 05ced83 into master Jan 19, 2024
11 checks passed

tadeohepperle deleted the tadeohepperle/cli-support-runtime-apis branch January 19, 2024 16:57

tadeohepperle mentioned this pull request Jan 23, 2024

Prep for v0.34.0 release #1395

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration #1290

CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration #1290

tadeohepperle commented Nov 24, 2023

tadeohepperle commented Jan 18, 2024

lexnv Jan 19, 2024

tadeohepperle Jan 19, 2024

jsdw Jan 19, 2024

jsdw Jan 19, 2024

jsdw Jan 19, 2024

tadeohepperle Jan 19, 2024

jsdw Jan 19, 2024

jsdw Jan 19, 2024

tadeohepperle Jan 19, 2024

jsdw Jan 19, 2024

jsdw commented Jan 19, 2024

jsdw left a comment

		#[allow(non_snake_case)]
		let INPUT_VALUE: String = "<INPUT_VALUE>".blue().to_string();

CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration #1290

CLI Tool Explore Command: Runtime APIs, Events, Colorized outputs, scale-typegen integration #1290

Conversation

tadeohepperle commented Nov 24, 2023

Testing

tadeohepperle commented Jan 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw commented Jan 19, 2024

jsdw left a comment

Choose a reason for hiding this comment