Elastic scaling: runtime dependency tracking and enactment #3479

alindima · 2024-02-26T10:44:51Z

Changes needed to implement the runtime part of elastic scaling: #3131, #3132, #3202

Also fixes #3675

TODOs:

storage migration
optimise process_candidates from O(N^2)
drop backable candidates which form cycles
fix unit tests
add more unit tests
check the runtime APIs which use the pending availability storage. We need to expose all of them, see Add new staging Runtime API: candidates_pending_availability #3576
optimise the candidate selection. we're currently picking randomly until we satisfy the weight limit. we need to be smart about not breaking candidate chains while being fair to all paras - ParaInherent create: update apply_weight_limit wrt elastic scaling #3573

Relies on the changes made in #3233 in terms of the inclusion policy and the candidate ordering

Signed-off-by: alindima <alin@parity.io>

also no need to sort by core index any more

…. optimise process_candidates to be O(N)

sandreim

Logic looks good at first pass, but readability can certainly be improved.

I think we should also see if we can adjust apply_weight_limit candidate selection to account for elastic scaling.

polkadot/runtime/parachains/src/paras_inherent/mod.rs

…aling-runtime

sandreim · 2024-03-05T11:03:51Z

candidate_pending_availability runtime API is needed by collators. At first glance we might need to return all of them. ~~or maybe just the last one.~~

ordian

Overall looking good, left a couple of questions. Happy to approve once this is tested/burned-in.

ordian · 2024-03-06T04:15:38Z

polkadot/runtime/parachains/src/paras_inherent/mod.rs

-		// In `Enter` context (invoked during execution) there should be no backing votes from
-		// disabled validators because they should have been filtered out during inherent data
-		// preparation (`ProvideInherent` context). Abort in such cases.
-		if context == ProcessInherentDataContext::Enter {
-			ensure!(!votes_from_disabled_were_dropped, Error::<T>::BackedByDisabled);
-		}


why was this error removed? is it because it was merged into a generic CandidatesFilteredDuringExecution error? i liked the specificity of the previous errors more

I believe the original intention was to trade the specific errors for simplicity by using a catch all approach. I will look and see if we can keep it simple and have these errors specific or maybe logging these errors instead of returning them might achieve same.

that's right. I generally added debug logs to the filtering functions called in sanitize_backed_candidates whenever a candidate is filtered and the reason why it was dropped.

Indeed, checking that filtering filtered nothing at the outer most level is the most robust way to check.

polkadot/runtime/parachains/src/inclusion/mod.rs

polkadot/runtime/parachains/src/paras_inherent/mod.rs

polkadot/runtime/parachains/src/inclusion/mod.rs

alindima · 2024-03-18T12:41:10Z

bot bench polkadot-pallet --runtime=rococo --pallet=runtime_parachains::paras_inherent -v UPSTREAM_MERGE=n

command-bot · 2024-03-18T12:41:16Z

@alindima https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5559128 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::paras_inherent. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 3-24f8b8dd-8273-4919-8221-ad79448fa599 to cancel this command or bot cancel to cancel all commands in this pull request.

alindima · 2024-03-18T13:56:27Z

While testing the runtime upgrade locally with zombienet, I found out that after the runtime upgrade, due to #64, availability_cores runtime API panics (the new runtime API is called with unmigrated storage).

availability_cores being such a critical API in all many parts of consensus, it seems that candidate backing and inclusion is stalled until the next session (when it recovers). I'm not sure which subsystem causes this, but I'm thinking of modifying the runtime API such that it will know how to work with both versions of the pallet storage format (v0 and v1).

Other than this, the PR should be ready

…=rococo --target_dir=polkadot --pallet=runtime_parachains::paras_inherent

command-bot · 2024-03-18T13:59:02Z

@alindima Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5559128 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5559128/artifacts/download.

…aling-runtime

alindima · 2024-03-19T08:15:34Z

Fixed the runtime API panic caused by #64 and reran benchmarks for westend and rococo

polkadot/runtime/parachains/src/runtime_api_impl/v7.rs

eskimor

Rococo weights seem off. Westend look good.

polkadot/runtime/parachains/src/paras_inherent/mod.rs

polkadot/runtime/parachains/src/runtime_api_impl/v7.rs

polkadot/runtime/parachains/src/paras_inherent/tests.rs

polkadot/runtime/rococo/src/weights/runtime_parachains_paras_inherent.rs

polkadot/runtime/westend/src/weights/runtime_parachains_paras_inherent.rs

polkadot/runtime/parachains/src/paras_inherent/mod.rs

alindima · 2024-03-19T16:19:46Z

yeah, the weights for rococo are way off when comparing to the previous values. I'm betting that's because they were last updated in 2021. The difference for westend shows that there isn't a significant change

…calingMVP is enabled

eskimor

Great work @alindima ! I couldn't help it and still had a few nits, but it is good to go!

eskimor · 2024-03-20T14:15:02Z

polkadot/runtime/parachains/src/paras_inherent/mod.rs

+		let freed = freed_concluded
+			.into_iter()
+			.map(|(c, _hash)| (c, FreedReason::Concluded))
+			.chain(freed_disputed.into_iter().map(|core| (core, FreedReason::Concluded)))


Not introduced here, but a third enum variant Disputed would have done no harm 😶‍🌫️ (and also no need to fix it here)

eskimor · 2024-03-20T14:26:06Z

polkadot/runtime/parachains/src/paras_inherent/tests.rs

+		// Cores 1, 2 and 3 are being made available in this block. Propose 6 more candidates (one
+		// for each core) and check that the right ones are successfully backed and the old ones
+		// enacted.
+		let config = default_config();


Given that we are not even sharing initialization, why is this not a separate test case?

to avoid long and dubious test names :D that's arguably a bad reason but I didn't think too much about it

polkadot/runtime/parachains/src/paras_inherent/tests.rs

…aling-runtime

sandreim

Amazing work @alindima

…h#3479) Changes needed to implement the runtime part of elastic scaling: paritytech#3131, paritytech#3132, paritytech#3202 Also fixes paritytech#3675 TODOs: - [x] storage migration - [x] optimise process_candidates from O(N^2) - [x] drop backable candidates which form cycles - [x] fix unit tests - [x] add more unit tests - [x] check the runtime APIs which use the pending availability storage. We need to expose all of them, see paritytech#3576 - [x] optimise the candidate selection. we're currently picking randomly until we satisfy the weight limit. we need to be smart about not breaking candidate chains while being fair to all paras - paritytech#3573 Relies on the changes made in paritytech#3233 in terms of the inclusion policy and the candidate ordering --------- Signed-off-by: alindima <alin@parity.io> Co-authored-by: command-bot <> Co-authored-by: eskimor <eskimor@users.noreply.github.com>

Initial draft changes

442185e

alindima marked this pull request as draft February 26, 2024 10:44

alindima added T8-polkadot This PR/Issue is related to/affects the Polkadot network. I5-enhancement An additional feature request. labels Feb 26, 2024

alindima added 9 commits February 26, 2024 16:45

bugfixes

799fabe

Signed-off-by: alindima <alin@parity.io>

filter descendants of disputed candidates

9f3ba62

also no need to sort by core index any more

some simplifications

5b13eb7

assert that candidates of a para are sorted in chain dependency order…

ecc5088

…. optimise process_candidates to be O(N)

deduplicate some of the logic for freeing cores

0fb7b8c

update some comments

0ed6fc3

unify dropped candidates errors

0a08fa0

add more logs

9e51e20

remove some todos

4e154e6

sandreim reviewed Feb 28, 2024

View reviewed changes

alindima added 12 commits February 29, 2024 10:07

review comments

ccef35b

some more nits

3f67898

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

c4fedd7

…aling-runtime

add runtime migration to inclusion storage

58b4129

add migration tests

23a8d34

don't allow candidate cycles

e0d9dff

fix bug

0198fb3

make tests compile and make paras_inherent tests pass

267fa05

fix inclusion tests

7733b25

fix some more tests

c32f925

clippy

40e2933

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

e8f5d2c

…aling-runtime

sandreim mentioned this pull request Mar 5, 2024

ParaInherent create: update apply_weight_limit wrt elastic scaling #3573

Merged

sandreim mentioned this pull request Mar 5, 2024

Add new staging Runtime API: candidates_pending_availability #3576

Closed

ordian reviewed Mar 6, 2024

View reviewed changes

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

f440217

…=rococo --target_dir=polkadot --pallet=runtime_parachains::paras_inherent

alindima added 3 commits March 18, 2024 16:00

Merge branch 'master' into alindima/elastic-scaling-runtime

2999589

fix runtime API panic

a745a63

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

1b70b11

…aling-runtime

fix clippy

c3060bd

sandreim reviewed Mar 19, 2024

View reviewed changes

polkadot/runtime/parachains/src/runtime_api_impl/v7.rs Show resolved Hide resolved

eskimor reviewed Mar 19, 2024

View reviewed changes

polkadot/runtime/parachains/src/paras_inherent/mod.rs Outdated Show resolved Hide resolved

alindima added 4 commits March 20, 2024 13:04

map_candidates_to_cores: check core index for single core if ElasticS…

9f52b8d

…calingMVP is enabled

add a couple more test cases

c27de6a

review comment

e05693c

Merge branch 'master' into alindima/elastic-scaling-runtime

bb76778

eskimor approved these changes Mar 20, 2024

View reviewed changes

alindima added 2 commits March 21, 2024 10:40

improve test

91f705c

Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…

0c8dc20

…aling-runtime

sandreim mentioned this pull request Mar 21, 2024

Add new variant of persisted_validation_data runtime API for elastic scaling #3776

Open

sandreim approved these changes Mar 21, 2024

View reviewed changes

Merge branch 'master' into alindima/elastic-scaling-runtime

b8d2212

eskimor enabled auto-merge March 21, 2024 09:47

eskimor added this pull request to the merge queue Mar 21, 2024

Merged via the queue into master with commit 4842faf Mar 21, 2024
127 of 132 checks passed

eskimor deleted the alindima/elastic-scaling-runtime branch March 21, 2024 10:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic scaling: runtime dependency tracking and enactment #3479

Elastic scaling: runtime dependency tracking and enactment #3479

alindima commented Feb 26, 2024 •

edited

sandreim left a comment

sandreim commented Mar 5, 2024 •

edited

ordian left a comment

ordian Mar 6, 2024

sandreim Mar 6, 2024

alindima Mar 11, 2024

eskimor Mar 13, 2024

alindima commented Mar 18, 2024

command-bot bot commented Mar 18, 2024 •

edited

alindima commented Mar 18, 2024

command-bot bot commented Mar 18, 2024

alindima commented Mar 19, 2024

eskimor left a comment

alindima commented Mar 19, 2024

eskimor left a comment

eskimor Mar 20, 2024

eskimor Mar 20, 2024

alindima Mar 20, 2024

eskimor Mar 20, 2024

sandreim left a comment

Elastic scaling: runtime dependency tracking and enactment #3479

Elastic scaling: runtime dependency tracking and enactment #3479

Conversation

alindima commented Feb 26, 2024 • edited

sandreim left a comment

Choose a reason for hiding this comment

sandreim commented Mar 5, 2024 • edited

ordian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alindima commented Mar 18, 2024

command-bot bot commented Mar 18, 2024 • edited

alindima commented Mar 18, 2024

command-bot bot commented Mar 18, 2024

alindima commented Mar 19, 2024

eskimor left a comment

Choose a reason for hiding this comment

alindima commented Mar 19, 2024

eskimor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandreim left a comment

Choose a reason for hiding this comment

alindima commented Feb 26, 2024 •

edited

sandreim commented Mar 5, 2024 •

edited

command-bot bot commented Mar 18, 2024 •

edited