Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Asynchronous backing PR #2300

Merged
merged 41 commits into from Aug 18, 2023
Merged

Conversation

slumber
Copy link
Contributor

@slumber slumber commented Mar 9, 2023

This is the Cumulus companion to asynchronous backing.

There are 2 major changes introduced in this PR, which will be expanded upon in a blog post:

Consensus Hooks

The major change of this PR is to introduce a ConsensusHook to the pallet_parachain_system::Config of a parachain. This ConsensusHook is a pluggable component which allows a chain to manage its asynchronous backing backlog. It is enabled by adding the parameterized-consensus-hook feature to pallet-parachain-system in your runtime. There is an implementation of this for Aura within pallet-aura-ext.

Lookahead Collator for Aura

Cumulus consensus architecture has been overhauled, and the new "Lookahead" collator logic for Aura takes into account asynchronous backing. Further details on how to integrate this collator will follow in a dedicated post.

See paritytech/polkadot#3779, paritytech/polkadot#5022

slumber and others added 2 commits April 20, 2023 01:01
…ocks (#2438)

* unincluded segment draft

* read para head from storage proof

* read_para_head -> read_included_para_head

* Provide pub interface

* add errors

* fix unincluded segment update

* BlockTracker -> Ancestor

* add a dmp limit

* Read para head depending on the storage switch

* doc comments

* storage items docs

* add a sanity check on block initialize

* Check watermark

* append to the segment on block finalize

* Move segment update into set_validation_data

* Resolve para head todo

* option watermark

* fix comment

* Drop dmq check

* fix weight

* doc-comments on inherent invariant

* Remove TODO

* add todo

* primitives tests

* pallet tests

* doc comments
@slumber slumber force-pushed the slumber-async-backing-feature branch from 3e20e1a to 86b5d3d Compare May 9, 2023 21:32
rphmeier and others added 20 commits May 16, 2023 20:29
* refactor unincluded segment length into a ConsensusHook

* add docs

* refactor bandwidth_out calculation

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* test for limits from impl

* fmt

* make tests compile

* update comment

* uncomment test

* fix collator test by adding parent to state proof

* patch HRMP watermark rules for unincluded segment

* get consensus-common tests to pass, using unincluded segment

* fix unincluded segment tests

* get all tests passing

* fmt

* rustdoc CI

* aura-ext: limit the number of authored blocks per slot (#2551)

* aura_ext consensus hook

* reverse dependency

* include weight into hook

* fix tests

* remove stray println

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* fix test warning

* fix doc link

---------

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
…2594)

* handle goahead signal for unincluded segment

* doc comment

* add test
* implement `drop_processed_messages`

* drop messages based on relay parent number

* adjust tests

* drop changes to mqc

* fix comment

* drop test

* drop more dead code
…` logic (#2658)

* aura-ext: check slot in consensus hook

* convert relay chain slot

* Make relay chain slot duration generic

* use fixed velocity hook for pallets with aura

* purge timestamp inherent

* fix warning

* adjust runtime tests

* fix slots in tests

* Make `xcm-emulator` test pass for new consensus hook (#2722)

* add pallets on_initialize

* tests pass

* add AuraExt on_init

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
* CollationGenerationConfig closure is now optional

* fix test
* First pass

* fmt

* Added as default feature in tomls

* Changed to direct dependency feature

* Dealing with clippy error

* Update pallets/parachain-system/src/lib.rs

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

---------

Co-authored-by: asynchronous rob <rphmeier@gmail.com>
…cluded segment (#2948)

* mostly address 2471 with a bug introduced

* adjust relevant messaging state after computing total

* fmt

* max -> min

* fix test implementation of xcmp source

* add test

* fix test message sending logic

* fix + test

* add more to unincluded segment test

* fmt

---------

Co-authored-by: Chris Sosnin <chris125_@live.com>
…/ Polkadot-Parachain (#2864)

* add a comment

* refactor client/service utilities

* deprecate start_collator

* update parachain-template

* update test-service in the same way

* update polkadot-parachain crate

* fmt

* wire up new SubmitCollation message

* some runtime utilities for implementing unincluded segment runtime APIs

* allow parachains to configure their level of sybil-resistance when starting the network

* make aura-ext compile

* update to specify sybil resistance levels

* fmt

* specify relay chain slot duration in milliseconds

* update Aura to explicitly produce Send futures

also, make relay_chain_slot_duration a Duration

* add authoring duration to basic collator and document params

* integrate new basic collator into parachain-template

* remove assert_send used for testing

* basic-aura: only author when parent included

* update polkadot-parachain-bin

* fmt

* some fixes

* fixes

* add a RelayNumberMonotonicallyIncreases

* add a utility function for initializing subsystems

* some logging for timestamp adjustment

* fmt

* some fixes for lookahead collator

* add a log

* update `find_potential_parents` to account for sessions

* bound the loop

* restore & deprecate old start_collator and start_full_node functions.

* remove unnecessary await calls

* fix warning

* clippy

* more clippy

* remove unneeded logic

* ci
Co-authored-by: Marcin S. <marcin@bytedude.com>
@rphmeier rphmeier added D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed. and removed T5-parachains This PR/Issue is related to Parachains. labels Aug 18, 2023
Copy link
Member

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't reviewed deeply, but approving to unblock merge.

Don't forget to revert diener patches.

.unwrap();
let pay_to = Polkadot::account_id_of(ALICE);
let pay_amount = 9000;
// let asset_id: u32 = 1984;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why comment out if you put ignore attribute? is it not compiling?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI executes ignored tests

@paritytech-processbot
Copy link

Merge cancelled due to error. Error: Command 'Command { std: cd "/storage/repositories/cumulus" && "cargo" "update" "-v" "-p" "pallet-bounties:4.0.0-dev" "-p" "frame-remote-externalities:0.10.0-dev" "-p" "pallet-referenda:4.0.0-dev" "-p" "pallet-contracts-proc-macro:4.0.0-dev" "-p" "pallet-contracts-primitives:24.0.0" "-p" "sc-proposer-metrics:0.10.0-dev" "-p" "sp-consensus-babe:0.10.0-dev" "-p" "pallet-aura:4.0.0-dev" "-p" "pallet-scheduler:4.0.0-dev" "-p" "sc-consensus-aura:0.10.0-dev" "-p" "pallet-staking-reward-fn:4.0.0-dev" "-p" "sp-genesis-builder:0.1.0" "-p" "sc-allocator:4.1.0-dev" "-p" "sp-panic-handler:8.0.0" "-p" "substrate-test-client:2.0.1" "-p" "sp-rpc:6.0.0" "-p" "sc-network:0.10.0-dev" "-p" "sp-core-hashing-proc-macro:9.0.0" "-p" "pallet-nomination-pools-benchmarking:1.0.0" "-p" "sp-runtime-interface:17.0.0" "-p" "frame-election-provider-solution-type:4.0.0-dev" "-p" "frame-support-procedural-tools-derive:3.0.0" "-p" "pallet-preimage:4.0.0-dev" "-p" "sc-service:0.10.0-dev" "-p" "frame-system-benchmarking:4.0.0-dev" "-p" "mmr-gadget:4.0.0-dev" "-p" "pallet-vesting:4.0.0-dev" "-p" "pallet-staking-runtime-api:4.0.0-dev" "-p" "sc-consensus-grandpa-rpc:0.10.0-dev" "-p" "pallet-election-provider-multi-phase:4.0.0-dev" "-p" "frame-try-runtime:0.10.0-dev" "-p" "pallet-beefy:4.0.0-dev" "-p" "pallet-transaction-payment-rpc-runtime-api:4.0.0-dev" "-p" "pallet-transaction-payment:4.0.0-dev" "-p" "pallet-beefy-mmr:4.0.0-dev" "-p" "pallet-proxy:4.0.0-dev" "-p" "pallet-multisig:4.0.0-dev" "-p" "sp-staking:4.0.0-dev" "-p" "pallet-nis:4.0.0-dev" "-p" "sc-network-transactions:0.10.0-dev" "-p" "pallet-collective:4.0.0-dev" "-p" "sc-consensus-grandpa:0.10.0-dev" "-p" "sp-transaction-pool:4.0.0-dev" "-p" "pallet-bags-list:4.0.0-dev" "-p" "pallet-im-online:4.0.0-dev" "-p" "pallet-tips:4.0.0-dev" "-p" "frame-support-procedural:4.0.0-dev" "-p" "sp-weights:20.0.0" "-p" "pallet-whitelist:4.0.0-dev" "-p" "substrate-frame-rpc-system:4.0.0-dev" "-p" "pallet-recovery:4.0.0-dev" "-p" "fork-tree:3.0.0" "-p" "sc-keystore:4.0.0-dev" "-p" "sp-version:22.0.0" "-p" "sp-trie:22.0.0" "-p" "substrate-test-utils:4.0.0-dev" "-p" "sc-consensus-slots:0.10.0-dev" "-p" "sc-offchain:4.0.0-dev" "-p" "sc-executor-wasmtime:0.10.0-dev" "-p" "sc-tracing:4.0.0-dev" "-p" "sp-externalities:0.19.0" "-p" "sp-mmr-primitives:4.0.0-dev" "-p" "frame-benchmarking-cli:4.0.0-dev" "-p" "sp-statement-store:4.0.0-dev" "-p" "sc-network-common:0.10.0-dev" "-p" "pallet-election-provider-support-benchmarking:4.0.0-dev" "-p" "pallet-nfts-runtime-api:4.0.0-dev" "-p" "pallet-salary:4.0.0-dev" "-p" "sc-network-bitswap:0.10.0-dev" "-p" "sc-network-gossip:0.10.0-dev" "-p" "sp-core:21.0.0" "-p" "sc-executor:0.10.0-dev" "-p" "pallet-elections-phragmen:5.0.0-dev" "-p" "sc-sync-state-rpc:0.10.0-dev" "-p" "sp-std:8.0.0" "-p" "sp-npos-elections:4.0.0-dev" "-p" "pallet-timestamp:4.0.0-dev" "-p" "pallet-transaction-payment-rpc:4.0.0-dev" "-p" "sc-chain-spec:4.0.0-dev" "-p" "sp-storage:13.0.0" "-p" "pallet-staking-reward-curve:4.0.0-dev" "-p" "sp-transaction-storage-proof:4.0.0-dev" "-p" "pallet-asset-conversion:4.0.0-dev" "-p" "sp-inherents:4.0.0-dev" "-p" "frame-system-rpc-runtime-api:4.0.0-dev" "-p" "try-runtime-cli:0.10.0-dev" "-p" "sc-chain-spec-derive:4.0.0-dev" "-p" "sc-sysinfo:6.0.0-dev" "-p" "pallet-utility:4.0.0-dev" "-p" "pallet-membership:4.0.0-dev" "-p" "sc-client-db:0.10.0-dev" "-p" "sc-rpc-spec-v2:0.10.0-dev" "-p" "pallet-offences:4.0.0-dev" "-p" "substrate-state-trie-migration-rpc:4.0.0-dev" "-p" "sc-consensus-epochs:0.10.0-dev" "-p" "pallet-session-benchmarking:4.0.0-dev" "-p" "pallet-asset-tx-payment:4.0.0-dev" "-p" "sp-metadata-ir:0.1.0" "-p" "substrate-rpc-client:0.10.0-dev" "-p" "pallet-identity:4.0.0-dev" "-p" "sc-informant:0.10.0-dev" "-p" "sc-rpc-api:0.10.0-dev" "-p" "frame-election-provider-support:4.0.0-dev" "-p" "sc-storage-monitor:0.1.0" "-p" "frame-support:4.0.0-dev" "-p" "sc-telemetry:4.0.0-dev" "-p" "sp-consensus:0.10.0-dev" "-p" "sc-basic-authorship:0.10.0-dev" "-p" "sc-cli:0.10.0-dev" "-p" "sp-blockchain:4.0.0-dev" "-p" "sp-consensus-slots:0.10.0-dev" "-p" "frame-executive:4.0.0-dev" "-p" "pallet-message-queue:7.0.0-dev" "-p" "sp-maybe-compressed-blob:4.1.0-dev" "-p" "sc-network-light:0.10.0-dev" "-p" "sc-rpc-server:4.0.0-dev" "-p" "sp-application-crypto:23.0.0" "-p" "sc-authority-discovery:0.10.0-dev" "-p" "substrate-build-script-utils:3.0.0" "-p" "substrate-prometheus-endpoint:0.10.0-dev" "-p" "pallet-ranked-collective:4.0.0-dev" "-p" "sc-transaction-pool-api:4.0.0-dev" "-p" "sp-timestamp:4.0.0-dev" "-p" "pallet-contracts:4.0.0-dev" "-p" "pallet-fast-unstake:4.0.0-dev" "-p" "sp-state-machine:0.28.0" "-p" "sc-client-api:4.0.0-dev" "-p" "sp-keystore:0.27.0" "-p" "pallet-treasury:4.0.0-dev" "-p" "pallet-glutton:4.0.0-dev" "-p" "pallet-uniques:4.0.0-dev" "-p" "pallet-authorship:4.0.0-dev" "-p" "sc-rpc:4.0.0-dev" "-p" "sc-transaction-pool:4.0.0-dev" "-p" "sp-core-hashing:9.0.0" "-p" "sp-arithmetic:16.0.0" "-p" "pallet-balances:4.0.0-dev" "-p" "sp-api:4.0.0-dev" "-p" "sc-consensus-beefy:4.0.0-dev" "-p" "sp-authority-discovery:4.0.0-dev" "-p" "sp-database:4.0.0-dev" "-p" "sp-debug-derive:8.0.0" "-p" "pallet-session:4.0.0-dev" "-p" "sp-runtime:24.0.0" "-p" "sp-block-builder:4.0.0-dev" "-p" "sp-runtime-interface-proc-macro:11.0.0" "-p" "sc-consensus-babe:0.10.0-dev" "-p" "sp-version-proc-macro:8.0.0" "-p" "pallet-grandpa:4.0.0-dev" "-p" "frame-system:4.0.0-dev" "-p" "pallet-society:4.0.0-dev" "-p" "pallet-babe:4.0.0-dev" "-p" "sc-block-builder:0.10.0-dev" "-p" "sc-consensus:0.10.0-dev" "-p" "sp-consensus-grandpa:4.0.0-dev" "-p" "sp-tracing:10.0.0" "-p" "pallet-nomination-pools-runtime-api:1.0.0-dev" "-p" "frame-benchmarking:4.0.0-dev" "-p" "pallet-nfts:4.0.0-dev" "-p" "binary-merkle-tree:4.0.0-dev" "-p" "sc-executor-common:0.10.0-dev" "-p" "pallet-insecure-randomness-collective-flip:4.0.0-dev" "-p" "pallet-offences-benchmarking:4.0.0-dev" "-p" "sc-consensus-babe-rpc:0.10.0-dev" "-p" "pallet-sudo:4.0.0-dev" "-p" "sp-consensus-beefy:4.0.0-dev" "-p" "substrate-test-utils-derive:0.10.0-dev" "-p" "pallet-conviction-voting:4.0.0-dev" "-p" "pallet-core-fellowship:4.0.0-dev" "-p" "sc-state-db:0.10.0-dev" "-p" "sp-api-proc-macro:4.0.0-dev" "-p" "frame-support-procedural-tools:4.0.0-dev" "-p" "sp-io:23.0.0" "-p" "pallet-democracy:4.0.0-dev" "-p" "sp-offchain:4.0.0-dev" "-p" "pallet-asset-conversion-tx-payment:4.0.0-dev" "-p" "sp-wasm-interface:14.0.0" "-p" "sc-utils:4.0.0-dev" "-p" "sp-consensus-aura:0.10.0-dev" "-p" "sp-session:4.0.0-dev" "-p" "pallet-nomination-pools:1.0.0" "-p" "mmr-rpc:4.0.0-dev" "-p" "pallet-alliance:4.0.0-dev" "-p" "pallet-state-trie-migration:4.0.0-dev" "-p" "sp-keyring:24.0.0" "-p" "pallet-assets:4.0.0-dev" "-p" "sc-network-sync:0.10.0-dev" "-p" "pallet-staking:4.0.0-dev" "-p" "pallet-child-bounties:4.0.0-dev" "-p" "sc-tracing-proc-macro:4.0.0-dev" "-p" "pallet-mmr:4.0.0-dev" "-p" "sc-consensus-beefy-rpc:4.0.0-dev" "-p" "pallet-indices:4.0.0-dev" "-p" "substrate-wasm-builder:5.0.0-dev" "-p" "pallet-nft-fractionalization:4.0.0-dev" "-p" "pallet-authority-discovery:4.0.0-dev", kill_on_drop: false }' failed with status Some(101); output: Updating git repository https://github.com/paritytech/substrate
Updating crates.io index
Updating git repository https://github.com/w3f/ring-vrf
error: failed to select a version for polkadot-service.
... required by package cumulus-relay-chain-inprocess-interface v0.1.0 (/storage/repositories/cumulus/client/relay-chain-inprocess-interface)
... which satisfies path dependency cumulus-relay-chain-inprocess-interface (locked to 0.1.0) of package cumulus-client-network v0.1.0 (/storage/repositories/cumulus/client/network)
... which satisfies path dependency cumulus-client-network (locked to 0.1.0) of package cumulus-client-collator v0.1.0 (/storage/repositories/cumulus/client/collator)
... which satisfies path dependency cumulus-client-collator (locked to 0.1.0) of package cumulus-client-service v0.1.0 (/storage/repositories/cumulus/client/service)
... which satisfies path dependency cumulus-client-service (locked to 0.1.0) of package cumulus-test-service v0.1.0 (/storage/repositories/cumulus/test/service)
... which satisfies path dependency cumulus-test-service (locked to 0.1.0) of package cumulus-client-pov-recovery v0.1.0 (/storage/repositories/cumulus/client/pov-recovery)
... which satisfies path dependency cumulus-client-pov-recovery (locked to 0.1.0) of package cumulus-client-consensus-common v0.1.0 (/storage/repositories/cumulus/client/consensus/common)
... which satisfies path dependency cumulus-client-consensus-common (locked to 0.1.0) of package cumulus-client-consensus-relay-chain v0.1.0 (/storage/repositories/cumulus/client/consensus/relay-chain)
versions that meet the requirements * (locked to 0.9.43) are: 0.9.43

the package cumulus-relay-chain-inprocess-interface depends on polkadot-service, with features: network-protocol-staging but polkadot-service does not have these features.

failed to select a version for polkadot-service which could resolve this conflict

Comment on lines +9968 to +9974
version = "0.9.16"
source = "git+https://github.com/paritytech/polkadot?branch=master#4b7822adeb666ad33539c7ecf1e4518d04b3a90c"
dependencies = [
"bitvec 1.0.1",
"fatality",
"futures",
"parity-scale-codec 2.3.1",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these versions should be bumped in polkadot

@ordian
Copy link
Member

ordian commented Aug 18, 2023

bot merge

@paritytech-processbot
Copy link

Waiting for commit status.

@paritytech-processbot paritytech-processbot bot merged commit 3cfe24f into master Aug 18, 2023
42 checks passed
@paritytech-processbot paritytech-processbot bot deleted the slumber-async-backing-feature branch August 18, 2023 16:57
@muharem
Copy link
Contributor

muharem commented Aug 22, 2023

@slumber any reason why the integration tests were commented? plans to get them back into master?

@slumber
Copy link
Contributor Author

slumber commented Aug 22, 2023

@slumber any reason why the integration tests were commented? plans to get them back into master?

See #3027 #2703

They're commented out because CI executes tests marked with #[ignore] anyway

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. B1-note_worthy Changes should be noted in the release notes C1-low PR touches the given topic and has a low impact on builders. D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed. T0-node This PR/Issue is related to the topic “node”.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet