Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

derive Serialize and Deserialize for XCM primitives #4997

Closed

Conversation

stanly-johnson
Copy link
Contributor

resolves #4930

sandreim and others added 30 commits January 10, 2022 11:00
* Remove sleep and use polkadot test service

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* updates

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix other tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Run metrics tests separately

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* copy some substrate utilities

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update runtime metric test

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Remove sleep from cli tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Polkadot companion for Substrate#10463 (paritytech#4519)

* Grandpa and Beefy protocol names include chain id

Signed-off-by: acatangiu <adrian@parity.io>

* chain_spec: include fork id

* use simplified protocol name

* fix after merge

* avoid using hash default, even for protocol names

* update lockfile for substrate

Co-authored-by: parity-processbot <>

* configuration: Update upgrade validation delay doc (paritytech#4662)

* typo

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo lock

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* use testnet profile

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Don't run with runtime-benchmark feature

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* conditional compile up one level

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Better docs on non initialized state.

* Document better what is happening.

* More precise errors.

* cargo fmt
* CI: remove rococo from CI

* CI: typo

* CI: another typo

Co-authored-by: CI system <>
* Move XCM runtime configurations into their own files

* Update copyright year

* Fix compilation errors

* Import XCM types in westend runtime unit tests
…ch#4652)

* Switch types to use `RuntimeDebug` instead of gated `Debug`

This is useful for when you want to debug in wasm and enable the
`force-debug` feature of `sp-debug-derive`.

* Fixes

* 🤦
* fix tests

* update lockfile for substrate

Co-authored-by: parity-processbot <>
* Add some docs to prevent a time loop.

* Review remarks.
…ack (paritytech#4705)

* approval-voting: add more logs

* approval-voting: query finalized block on startup and increase look back
* approval-voting: add more logs

* approval-voting: query finalized block on startup and increase look back

* runtime-api: do not cache None SessionInfo
* cargo: sync workspace members with dependencies

* Remove bridges stuff

* Fix

Co-authored-by: Bastian Köcher <info@kchr.de>
* Check authority status on active leaves update

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo changes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add metric for authority status

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Revert "Fix tests"

This reverts commit 5bd56bb.

* Revert "cargo changes"

This reverts commit ffea18f.

* Revert "Check authority status on active leaves update"

This reverts commit 55a30ac.

* Test fixups

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* undo damage

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* dont update status on runtime errors

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix inconsistency

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Dont derive primitive Default

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* add dummy_session_info helper

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* unset parachain validator status if no longer authority

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* damn

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* 🤦

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
In this PR, paras module emit runtime events on certain PVF pre-checking
related conditions.

Specifically, there are 3 new events in the paras module:

1. PvfCheckStarted
2. PvfCheckAccepted
3. PvfCheckRejected

All of those have identifiers for the parachain that triggered the PVF
pre-checking and the validation code that goes through the pre-checking.

The mechanics of those are as follows. Each time a new PVF is added, be
it due to onboarding or upgrading, the `PvfCheckStarted` will be
triggered. If another parachain triggers a pre-checking process for the
validation code which is already being pre-checked, another
`PvfCheckStarted` event will be triggered with the corresponding para
id.

When the PVF pre-checking voting for a PVF was finished, several
`PvfCheckAccepted/Rejected` events will be triggered: one for each para id that
was subscribed to this check (i.e. was a "cause" for it).

If the PVF pre-checking is disabled, then one can still expect these
events to be fired. Since insta PVF approval is syncronous, the
`PvfCheckStarted` will be followed by the `PvfCheckAccepted` with the
same validation code and para id.

If somebody is interested in following validation code changes for a PVF
of a parachain, they would need to subscribe to those events. I did not
supply the topics for the events, since I am not sure if that's needed
or will be used, but they can be added later if needed.
* add fast-runtime feature for reduced session times

* make democracy periods fast on fast-runtime

* propagate fast-runtime feature through cargo.toml files

* add fast motion and term durations to Kusama

* Update runtime/westend/Cargo.toml

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* set session time to 2 minutes to avoid block production issues

* formatting

* update Substrate

* set democracy fast periods back to 1min

* set launch period and enactment period to 1 block in fast-runtime

* remove unnecessary westend period configs

* add prod_or_test macro to allow specifying prod, test and env values for parameter types

* move prod_or_test macro into common module and use it consistently

* rename macro to prod_or_fast

* cargo +nightly fmt

* bump impl_versions

* newline

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* add note that env variable is evaluated at compile time

* newline

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* newline

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* cargo fmt

* impl_version: 0

* impl_version: 0

* use prod_or_fast macro for LeasePeriod and LeaseOffset

* use prod_or_fast macro in WND and ROC constants

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
* Change EraIndex import path: companion #10671

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
if the PVF pre-checking is disabled the runtime dispatchable will reject
any attempts of submission. This is also concern the unsigned tx
validation.

Right now, the `include_pvf_check_statement` dispatchable is effectively
uncallable because of the weight set to the maximum value. If we were to
benchmark it, it would become includable in a block, but since there
will be no active votes, the dispatchable won't do anything.

However, it will execute some code, like signature validation and
querying some storage entries. To be completely safe, we can bail out
early if the `pvf_checking_enabled` config is disabled. That's what this
PR does.
It also moves some simple scripts directly into the .gitlab.yaml file.
* Make `pallet_xcm::execute` supply the proper weight credit

* Use weight correction

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* pvf-checker-subsystem: metrics

This commits adds metrics to the PVF pre-checking subsystem.

* Apply suggestions from code review

Co-authored-by: sandreim <54316454+sandreim@users.noreply.github.com>

Co-authored-by: sandreim <54316454+sandreim@users.noreply.github.com>
coderobe and others added 12 commits February 24, 2022 16:56
* scripts: fix polkadot_injected_release gpg errors

* ci: publish-docker-release: remove keyserver arg

* scripts: join RUNs in polkadot_injected_release
* ci: add manual dispatch docker publish workflow

* ci: fix date in dispatch trigger

* ci: avoid specifying gpg keyserver in multiple places
* Updating dependencies

* Adapting code to scale v3

* Upgrade bitvec to 1.0.0

* Fix bitvec arithmetics

* Update Cargo.lock

* Update sp-io

* Fixing the build

* Yanked scale-info 2.0.0

Co-authored-by: Bastian Köcher <info@kchr.de>
* Try to fix disputes test

* Enable all native runtimes
* seed commit for fatality based errors

* fatality

* first draft of fatality

* cleanup

* differnt approach

* simplify

* first working version for enums, with documentation

* add split

* fix simple split test case

* extend README.md

* update fatality impl

* make tests passed

* apply fatality to first subsystem

* fatality fixes

* use fatality in a subsystem

* fix subsystem�g�����

* fixup proc macro

* fix/test: log::*! do not execute when log handler is missing

* fix spelling

* rename Runtime2 to something sane

* allow nested split with `forward` annotations

* add free license

* enable and fixup all tests

* use external fatality

Makes this more reviewable.

* bump fatality dep

Avoid duplicate expander compilations.

* migrate availability distribution

* more fatality usage

* chore: bump fatality to 0.0.6

* fixup remaining subsystems

* chore: fmt

* make cargo spellcheck happy

* remove single instance of `#[fatal(false)]`

* last quality sweep

* fixup
* Add storage benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
…#4972)

* Add a simple metric for statements out-of-view

* Avoid repeated out-of-view peer reputation change messages

* Log reporting status

* Address review comments

* Use counter to store a number of unexpected messages from a peer

* Distinguish different unexpected statements in the metrics

* Fix labels cardinality

* Rename metric name to `statements_unexpected`

* Move metrics to a separate unit, avoid unnecessary enum

* Prefer specific methods in lieu of public constants
Bumps [color-eyre](https://github.com/yaahc/color-eyre) from 0.6.0 to 0.6.1.
- [Release notes](https://github.com/yaahc/color-eyre/releases)
- [Changelog](https://github.com/yaahc/color-eyre/blob/master/CHANGELOG.md)
- [Commits](eyre-rs/color-eyre@v0.6.0...v0.6.1)

---
updated-dependencies:
- dependency-name: color-eyre
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Claim grounds, not working yet.

* still wip, nothing to see

* Module visibility problem

* Add benchmarks for all parachain-enabled chains

* Add missing mock weights

* Fix mock weight info

* Add max-weight to extrinsic weight

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_ump.rs

* sorting declarations

* Use a maximum size message to benchmark worst-case

* remove warning

* Fix msg size overflow in tests

* Use benchmarked weight in PostInfo

* Update runtime/parachains/src/ump.rs

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>

* Delete obsolete file

* Compile fix

* Fix spelling

* Add benchmark for `UmpSink::process_upward_message`

It benchmarks the time to process a message, mainly the size-dependent
decoding time. This is used interally for refunding weight from
`service_overweight`

* Fix warning

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_ump.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_ump.rs

* Update runtime/parachains/src/ump.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Give mock weights for tests a name instead of ()

* Add benchmark for `initializer_on_new_session` aka.
`perform_outgoing_para_cleanup`

* Remove unnescessary pub specifier and change folding to a for loop

suggestions by @pepyakin

* Update runtime/parachains/src/ump.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update runtime/parachains/src/ump.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Add comment on weight calculation and expected slight overestimation

* Introduce a size bound constant for upward messages

* Use KiB not kB for message size bound

* Enforce max-message size bound

* Use a const for upward message size bound.

* Decrease max_upward_message_sizes to 50KiB as suggested by @pepyakin

* Fix creating overweight messages

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_ump.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_ump.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_ump.rs

* Remove unused variable

* Rename benchmark function to the function actually benchmarked and some
code cleanup

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_ump.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_ump.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::ump --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_ump.rs

* Better variable naming

* Fix build

* Fix build

* Fix some remarks from @pepyakin

Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
* Manage new `sr25519::Signature::from_slice` fallible constructor

Constructor now returns an `Option`

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
@bkchr
Copy link
Member

bkchr commented Feb 27, 2022

@stanly-johnson did you even test this? As you see the entire CI is failing.

Copy link
Member

@bkchr bkchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should probably also directly target the XCMV3 branch. WDYT @KiChjang ?

Comment on lines +35 to +36
#[cfg(feature = "std")]
use serde::{Deserialize, Serialize};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg(feature = "std")]
use serde::{Deserialize, Serialize};


/// A general identifier for an instance of a non-fungible asset class.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, Debug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -97,6 +100,7 @@ impl From<Vec<u8>> for AssetInstance {

/// Classification of an asset being concrete or abstract.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Debug, Encode, Decode, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -146,6 +150,7 @@ impl AssetId {

/// Classification of whether an asset is fungible or not, along with a mandatory amount or instance.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Debug, Encode, Decode, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -173,6 +178,7 @@ impl<T: Into<AssetInstance>> From<T> for Fungibility {
}

#[derive(Clone, Eq, PartialEq, Debug, Encode, Decode, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

Comment on lines +22 to +23
#[cfg(feature = "std")]
use serde::{Deserialize, Serialize};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg(feature = "std")]
use serde::{Deserialize, Serialize};


/// A global identifier of an account-bearing consensus system.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, Debug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -35,6 +38,7 @@ pub enum NetworkId {

/// An identifier of a pluralistic body.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, Debug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -56,6 +60,7 @@ pub enum BodyId {

/// A part of a pluralistic body.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, Debug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@@ -103,6 +108,7 @@ impl BodyPart {
///
/// Each item assumes a pre-existing location as its context and is defined in terms of it.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, Debug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", derive(serde::Serialize, serde::Deserialize))]

@KiChjang
Copy link
Contributor

KiChjang commented Mar 2, 2022

It should probably also directly target the XCMV3 branch. WDYT @KiChjang ?

Yes, this should happen as well, so that Gav may have a chance to look at it and be aware of the changes made here.

@stanly-johnson stanly-johnson changed the base branch from master to xcm-v3 March 6, 2022 06:03
@stanly-johnson stanly-johnson changed the base branch from xcm-v3 to master March 6, 2022 06:03
@stanly-johnson stanly-johnson changed the base branch from master to xcm-v3 March 6, 2022 06:22
@stanly-johnson
Copy link
Contributor Author

messed up the rebase, replaced by #5036

@stanly-johnson stanly-johnson deleted the derive_serialize branch March 6, 2022 06:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

derive Serialize and Deserialize for XCM primitives