V15 Metadata: Support accessing custom types #1106

tadeohepperle · 2023-08-03T12:53:59Z

Custom types are now available from OfflineClientT like this:

let api: OfflineClient<Config>;
let ty = api.custom_types().get("Foo")?;
let foo: Foo = ty.as_type()?;

lexnv · 2023-08-08T07:35:27Z

subxt/src/lib.rs

@@ -5,7 +5,7 @@
 //! Subxt is a library for interacting with Substrate based nodes. Using it looks something like this:
 //!
 //! ```rust,ignore
-#![doc = include_str!("../examples/tx_basic.rs")]
+#![doc = include_str ! ("../examples/tx_basic.rs")]


nit: Is this intended?

lexnv · 2023-08-08T07:39:12Z

subxt/src/custom_types/mod.rs

+
+impl<'a> CustomValueMetadata<'a> {
+    /// the scale encoded value
+    pub fn encoded(&self) -> &[u8] {


nit: We could name this value and update the documentation to say The custom value associated with the type, since developers are free to place any binary representation/value here

I wouldn't call it value, because that overlaps with a lot of naming we use where value returns a scale_value::Value. encoded() is consistent with other areas of the API at least (.bytes() would also work to me, to signal that it's quite arbitrary and may not necessarily be a scale encoded value)

lexnv

Lovely PR and useful to expose those custom fields to users!
Do you think it would be worth exposing the custom types in the codegen as well?

jsdw · 2023-08-08T08:46:11Z

Do you think it would be worth exposing the custom types in the codegen as well?

That's an interesting thought. Being quite arbitrary, some of the types and values may not line up (and so codegen should be lenient and just ignore any errors), and the string names are also arbitrary and need not be valid idents or anything, but one could expose codegen for the cases where things appear to line up. It'd be cool to be able to do let query = runtime::custom().foo(); api.custom().at(&query) to access things in custom, decoded to some concrete type (if possible).

I'll raise an issue so we can consider this as a follow-up

jsdw · 2023-08-08T09:31:14Z

metadata/src/lib.rs

+    pub fn custom_metadata(&self) -> &CustomMetadata<PortableForm> {
+        &self.custom
+    }


I think we should hide the internals (frame-metadata stuff) a little better here and have an api more like:

let custom = metadata.custom(); let Some(value) = custom.get("Foo") else { return }; let type_id: u32 = value.ty(); let data: &[u8] = value.bytes(); // so maybe thats implemented something like: pub struct CustomMetadata { // We can leave the frame_metadata map as is: values: BTreeMap<String, frame_metadata::CustomValueMetadata<T>> } pub struct CustomMetadataValue<'a> { type_id: u32, data: &'a Vec<u8> } impl CustomMetadata { pub fn get(&'a self, &str) -> Option<CustomMetadataValue<'a>> } impl CustomMetadataValue { pub fn ty(&self) -> u32; pub fn bytes(&Self) -> &[u8] }

jsdw · 2023-08-08T09:32:16Z

subxt/src/client/online_client.rs

+    /// Access custom types.
+    pub fn custom_types(&self) -> CustomTypes {


It's really the values that we are accessing (and the values happen to come with type info) so maybe custom_values -> CustomValues makes more sense?

subxt/src/custom_types/mod.rs

jsdw · 2023-08-09T14:52:25Z

metadata/src/from_into/v15.rs

@@ -17,6 +17,7 @@ use std::collections::HashMap;
 // Converting from V15 metadata into our Subxt repr.
 mod from_v15 {
    use super::*;
+    use crate::CustomMetadata;


We don't need this for all of the other types?

jsdw · 2023-08-09T14:52:55Z

metadata/src/lib.rs

+/// Metadata of custom types with custom values, basically the same as `frame_metadata::v15::CustomMetadata<PortableForm>>`.
+#[derive(Debug, Clone)]
+pub struct CustomMetadata {
+    pub(crate) map: BTreeMap<String, frame_metadata::v15::CustomValueMetadata<PortableForm>>,


I don't think the pub(crate) is necessary (since sub-modules can access stuff in parent ones anyway, and it's not needed anywhere else)

jsdw · 2023-08-09T14:53:50Z

metadata/src/lib.rs

+    pub fn bytes(&self) -> &[u8] {
+        &self.data
+    }


Suggested change

pub fn bytes(&self) -> &[u8] {

&self.data

}

pub fn bytes(&self) -> &'a [u8] {

&self.data

}

Just to tie the lifetime to that of the bytes and not that of this CustomMetadataValue instance

Oh yeah true, nice catch!

jsdw · 2023-08-09T14:55:47Z

subxt/src/custom_values/custom_value_adress.rs

Typo in the file name, should be custom_value_address.rs :)

jsdw · 2023-08-09T15:04:03Z

subxt/src/custom_values/custom_value_adress.rs

+    fn name(&self) -> &str;
+}
+
+impl CustomValueAddress for str {


Maybe for consistency we just impl this on &str so that we don't need the + ?Sized bound below? I'm a bit torn; it doesn't really matter either way afaik :)

I wanted to use str directly, because of this interface:

at<Address: CustomValueAddress + ?Sized>(&self, address: &Address)

When I used &str in the trait implementation, this is expecting &&str to be passed as address. And then it cannot be easily used as e.g. custom_value_client.at("Foo") anymore.

jsdw · 2023-08-09T15:05:28Z

subxt/src/custom_values/custom_values_client.rs

+                map: {
+                    let mut m = BTreeMap::new();
+                    m.insert("Person".to_string(), person_value_metadata);
+                    m
+                },


Just FYI a nice alternate way to declare this is:

Suggested change

map: {

let mut m = BTreeMap::new();

m.insert("Person".to_string(), person_value_metadata);

m

},

map: BTreeMap::from_iter([

("Person".to_string(), person_value_metadata)

]),

jsdw · 2023-08-09T15:06:41Z

Just some minor comments but looks great overall; nice one!

…type-map

jsdw · 2023-08-11T08:37:23Z

subxt/src/book/usage/custom_values.rs

+//!
+//! Note: Names of custom values are converted to __snake_case__ to produce a valid function name during code generation.
+//! If there are multiple values where the names would be equal when converted to __snake_case__, functions might not be statically generated for some of them, because of naming conflicts.
+//! Make sure names in the custom values of your metadata differ significantly.


Suggested change

//! Make sure names in the custom values of your metadata differ significantly.

Just because I think it's covered above, and users often won't have the ability to influence chain metadata anyway :)

jsdw · 2023-08-11T08:38:13Z

subxt/src/book/usage/custom_values.rs

+//! let api = OnlineClient::<PolkadotConfig>::new().await?;
+//! let custom_value_client = api.custom_values();
+//!
+//! // Now the `at()` function already decodes the value into the Foo type:


Suggested change

//! // Now the `at()` function already decodes the value into the Foo type:

//! // Now, the `at()` function decodes the value into the correct type:

jsdw · 2023-08-11T08:38:30Z

subxt/src/book/usage/custom_values.rs

Lovely docs; nice one!

Two small things you'll need to do to make tests pass:

import metadata from the right location (look at other doc examples for this)

wrap the doc stuff in a hidden #[tokio::main] async fn main() thing (use # to hide this; check out other examples to see :))

Lovely docs indeed! 🚀

jsdw · 2023-08-11T08:40:46Z

subxt/src/book/usage/custom_values.rs

+//!
+//! ```
+//!
+//! Alternatively we also provide a statically generated api for custom values:


I guess the logic for this is a followup PR, but as long as we don't do another release before the static stuff merges too then it's ok to leave the docs here :)

jsdw

Looks great to me!

lexnv

LGTM! Amazing job here!

tadeohepperle added 2 commits August 3, 2023 01:58

work in progress

09a045f

add custom types access

e2af2cb

tadeohepperle requested a review from a team as a code owner August 3, 2023 12:54

lexnv reviewed Aug 8, 2023

View reviewed changes

jsdw mentioned this pull request Aug 8, 2023

Custom metadata values; consider adding codegen #1113

Closed

jsdw reviewed Aug 8, 2023

View reviewed changes

subxt/src/custom_types/mod.rs Outdated Show resolved Hide resolved

tadeohepperle added 3 commits August 8, 2023 13:23

nit

72ad54b

Merge branch 'master' into tadeo-hepperle-v15-custom-type-map

8a9e76a

custom values client

50da61c

jsdw reviewed Aug 9, 2023

View reviewed changes

subxt/src/custom_values/custom_value_adress.rs Outdated

Copy link

Collaborator

jsdw Aug 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in the file name, should be custom_value_address.rs :)

jsdw reviewed Aug 9, 2023

View reviewed changes

tadeohepperle added 6 commits August 9, 2023 18:31

adjust light client

e8f2140

Merge remote-tracking branch 'origin' into tadeo-hepperle-v15-custom-…

00cbee4

…type-map

adjust doc comments

f286b26

adjust book for custom values in code gen

2f83d36

format and check docs

57be757

Merge branch 'master' into tadeo-hepperle-v15-custom-type-map

4a9000a

jsdw reviewed Aug 11, 2023

View reviewed changes

jsdw approved these changes Aug 11, 2023

View reviewed changes

use ignore in docs in book

922fee2

lexnv approved these changes Aug 11, 2023

View reviewed changes

tadeohepperle merged commit 8ba113f into master Aug 11, 2023
9 checks passed

tadeohepperle deleted the tadeo-hepperle-v15-custom-type-map branch August 11, 2023 12:49

jsdw mentioned this pull request Sep 27, 2023

Prep to release 0.32.0 #1178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V15 Metadata: Support accessing custom types #1106

V15 Metadata: Support accessing custom types #1106

tadeohepperle commented Aug 3, 2023

lexnv Aug 8, 2023

lexnv Aug 8, 2023

jsdw Aug 8, 2023

lexnv left a comment

jsdw commented Aug 8, 2023

jsdw Aug 8, 2023

jsdw Aug 8, 2023

jsdw Aug 9, 2023

jsdw Aug 9, 2023

jsdw Aug 9, 2023

tadeohepperle Aug 9, 2023

jsdw Aug 9, 2023

jsdw Aug 9, 2023

tadeohepperle Aug 9, 2023 •

edited

Loading

jsdw Aug 9, 2023

jsdw commented Aug 9, 2023

jsdw Aug 11, 2023

jsdw Aug 11, 2023

jsdw Aug 11, 2023

jsdw Aug 11, 2023

lexnv Aug 11, 2023

jsdw Aug 11, 2023

jsdw left a comment

lexnv left a comment

		/// Access custom types.
		pub fn custom_types(&self) -> CustomTypes {

	//! // Now the `at()` function already decodes the value into the Foo type:
	//! // Now, the `at()` function decodes the value into the correct type:

V15 Metadata: Support accessing custom types #1106

V15 Metadata: Support accessing custom types #1106

Conversation

tadeohepperle commented Aug 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lexnv left a comment

Choose a reason for hiding this comment

jsdw commented Aug 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tadeohepperle Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw commented Aug 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw left a comment

Choose a reason for hiding this comment

lexnv left a comment

Choose a reason for hiding this comment

tadeohepperle Aug 9, 2023 •

edited

Loading