Enrich metadata with type information

[ascjones]

⚠️ Breaking Metadata Change ⚠️

Closes #917 👀,

polkadot companion: paritytech/polkadot#3336

Migration guide: https://gist.github.com/ascjones/0d81a4c44e84cacd9f714cd34a6de823

Overview

• Replaces in tree frame-metadata crate with https://github.com/paritytech/frame-metadata/
• New v14 metadata structure defined here, which contains type information using scale-info
• Annotates all SCALE encoded types with TypeInfo derive which will generate the static type information to be provided in the metadata.

See the latest generated metadata json for the default node-runtime here. This was generated by running the substrate node on this branch and then running https://github.com/ascjones/subsee to download and decode -> json serialize.

A prototype (Rust only for now) code generator exists here: https://github.com/ascjones/chameleon to verify whether there is enough information in the metadata to generate all types.

todo:

• Generate errors metadata
• Restore and fix all commented out tests
• Figure out how to handle the existing DefaultByteGetter pattern for storage/constants for the new metadata.
• scale-info 0.7 release containing all substrate compatibility features
  • update parity-common and frame-metadata to use this release
• Decide how to handle custom (non derived) codec encoding (see todos).
• Are the all the (required for scale-info type registration) vec! allocations okay which replace existing static array refs?
• Clean up and unify namespacing of frame-metadata and scale-info types, e.g. remove all the v13s and add some imports to remove fully qualified types, which are there primarily for easy grepping during development.
• Polkadot companion
• Revisit handling of PhantomData in scale-info as it can in some cases require an unnecessary trait bound for the generic parameter. Can just skip those fields, or just revisit the approach, see https://github.com/paritytech/scale-info/blob/f0f689be810fc221a2aa9f015c0b0e1cd35fdd55/src/ty/mod.rs#L425
• Benchmark:
  • Code size: by how much does this increase the size of the resulting wasm blob? If significantly we should consider storing the metadata somewhere else other than the runtime (see Rich type information for runtime metadata using `scale-info` #8370 (comment))
  • The size of the encoded metadata itself, the result of state_getMetadata, is there a benefit to oregenerating/compressing it? Relates to previous point.
  • Compilation times: how does having the extra type metadata affect the overall substrate compilation times? If a considerable slowdown then should we consider feature gating the metadata generation?
• Replace existing frame-metadata crate with https://github.com/paritytech/frame-metadata
  • Remove existing crate and replace all usages
  • Publish frame-metadata to crates.io and replace git dependecies with that
• Verify encoding and decoding of substrate calls/events/storage/constants using the metadata, e.g. using https://github.com/ascjones/chameleon
• Note follow ups from review comments, possibly create an issue for them.

Follow ups:

• avoid explicit OpaqueMetadata::new(Runtime::metadata().into()) in runtime metadata API: Enrich metadata with type information #8615 (comment)

[olanod]

Would paritytech/frame-metadata#6 be to big/out of scope to be part of this new version?

[ascjones]

Not sure, it certainly would be of enormous benefit in terms of the size of the metadata.

The consideration would be how to generate/distribute the version of the metadata which includes the docs, perhaps could be embedded with the clients which use it e.g. polkadot.js. So needs some thinking about which I've not done yet.

[bkchr]

* Figure out how to handle the existing `DefaultByteGetter` pattern for storage/constants for the new metadata.

Just kill it. This never brought any benefit.

[olanod]

Not sure, it certainly would be of enormous benefit in terms of the size of the metadata.

The consideration would be how to generate/distribute the version of the metadata which includes the docs, perhaps could be embedded with the clients which use it e.g. polkadot.js. So needs some thinking about which I've not done yet.

How sensible is it to introduce new RPC calls, e.g. state_getMetadataCompact? and what about encoding of the results, is it set in stone somewhere that binary blobs have to be encoded in hex and no base64 for example? would be nicer if compact means is even more compact.

[real-felix]

I've created a newtype as pub struct Set<T>(Vec<T>);, but when I try to use it in a storage, it is marked as Set, without the generic information. Thus I cannot use it with several different generic parameters in polkadot.js.

Is it related to this issue? Is it soon to be solved?