Benchmark Regression Tracking Issue

[shawntabrizi]

In #5097, it was shown some benchmarks were not working.

• runtime/kusama/src/weights/frame_system.rs
• runtime/kusama/src/weights/pallet_collective_council.rs
• runtime/kusama/src/weights/pallet_collective_technical_committee.rs
• runtime/kusama/src/weights/pallet_election_provider_multi_phase.rs
• runtime/kusama/src/weights/pallet_elections_phragmen.rs
• runtime/kusama/src/weights/runtime_common_crowdloan.rs
• runtime/polkadot/src/weights/frame_system.rs
• runtime/polkadot/src/weights/pallet_collective_council.rs
• runtime/polkadot/src/weights/pallet_collective_technical_committee.rs
• runtime/polkadot/src/weights/pallet_election_provider_multi_phase.rs
• runtime/polkadot/src/weights/runtime_common_auctions.rs
• runtime/polkadot/src/weights/runtime_common_crowdloan.rs
• runtime/polkadot/src/weights/runtime_common_slots.rs
• runtime/polkadot/src/weights/runtime_parachains_hrmp.rs
• runtime/westend/src/weights/frame_system.rs
• runtime/westend/src/weights/pallet_election_provider_multi_phase.rs
• runtime/westend/src/weights/runtime_common_crowdloan.rs

This issue will track making sure each of these works on the appropriate runtime, and work on a fix if not.

[coderobe]

frame_system benchmarks did end up working via benchapp, it's possible that scripts/run_benches_for_runtime.sh just failed to trigger those.

As for the rest, here are some logs for benchmark runs of the main runtimes, might be of use:
https://gitlab.parity.io/parity/polkadot/-/jobs/1439417/raw
https://gitlab.parity.io/parity/polkadot/-/jobs/1439418/raw
https://gitlab.parity.io/parity/polkadot/-/jobs/1439419/raw

[shawntabrizi]

thank you @coderobe

[ggwpez]

The runtime_parachains::ump benchmark still reports Invalid input: UnknownMessageIndex.
Do you know what that means @hirschenberger? We are currently trying to fix some regressions.
cc @KiChjang

[hirschenberger]

The runtime_parachains::ump benchmark still reports Invalid input: UnknownMessageIndex. Do you know what that means @hirschenberger? We are currently trying to fix some regressions. cc @KiChjang

Is it in the service_overweight benchmark? It means that the message has not been overweight and was not enqueued to the overweight queue. That might be bc. of some bound-value changes.

So the create_message_overweight function may not have produced an overweight message.

Is it urgent? I can have a look on it at the weekend.

[ggwpez]

Okay thanks for your quick reply!

Is it urgent? I can have a look on it at the weekend.

We can do the next release still with the old weight numbers I think, so that is no problem.

[hirschenberger]

@ggwpez @shawntabrizi Ah, we're talking about polkadot benchmarks.

Ump-benchmarks never worked for polkadot because the polkadot runtime is configured to use a mock UmpSinkthat does not do anything see here and here.

When I started developing the benchmarks in pre-parachain times, I just skipped polkadot bc. it seemed to me UMP is not ready for primetime in polkadot.

[shawntabrizi]

AFAIK this is completed. cc @coderobe