Small optimisation to `--profile dev` wasm builds #1851

liamaharon · 2023-10-11T12:52:58Z

wasm-builder was adjusted to default to building wasm blobs in release mode even when cargo is in debug because debug wasm is too slow.

A side effect of this was .compact and .compact.compressed getting built when the dev is running build in debug, adding ~5s to the build time of every wasm runtime.

I think it's reasonable to assume if the dev is running debug build they want to optimise speed and do not care about the size of the wasm binary. Compacting a blob has negligible impact on its actual performance.

In this PR, I adjusted the behavior of the wasm builder so it does not produce .compact or .compact.compressed wasm when the user is running in debug. The builder will continue to produce the bloaty wasm in release mode unless it is overriden with an env var.

As suggested by @koute in review, also refactored the maybe_compact_wasm_and_copy_blobs into multiple funuctions, and renamed things to better support RISC-V in the future.

There is no T-runtime label so @KiChjang told me to put T1-FRAME :)

substrate/utils/wasm-builder/src/wasm_project.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

substrate/utils/wasm-builder/src/wasm_project.rs

* master: Removed TODO from test-case for hard-coded delivery fee estimation (#2042) Expose collection attributes from `Inspect` trait (#1914) `polkadot-parachain-primitives` should not depend on `frame-support`. (#1897) [testnet] Align testnet system parachain runtimes using `RelayTreasuryLocation` and `SystemParachains` in the same way (#2023) Sort the benchmarks before listing them (#2026) publish pallet-root-testing (#2017) Contracts: Add benchmarks to include files (#2022) Small optimisation to `--profile dev` wasm builds (#1851) basic-authorship: Improve time recording and logging (#2010) Application Crypto and BEEFY Support for paired (ECDSA,BLS) crypto (#1815) [ci] Run check-rust-feature-propagation in pr and master (#2012) Improve features dev-ex (#1831) Remove obsolete comment. (#2008)

* tsv-disabling: Removed TODO from test-case for hard-coded delivery fee estimation (#2042) Expose collection attributes from `Inspect` trait (#1914) `polkadot-parachain-primitives` should not depend on `frame-support`. (#1897) [testnet] Align testnet system parachain runtimes using `RelayTreasuryLocation` and `SystemParachains` in the same way (#2023) Sort the benchmarks before listing them (#2026) publish pallet-root-testing (#2017) Contracts: Add benchmarks to include files (#2022) Small optimisation to `--profile dev` wasm builds (#1851) basic-authorship: Improve time recording and logging (#2010) Application Crypto and BEEFY Support for paired (ECDSA,BLS) crypto (#1815) [ci] Run check-rust-feature-propagation in pr and master (#2012) Improve features dev-ex (#1831) Remove obsolete comment. (#2008)

* tsv-disabling: (36 commits) Removed TODO from test-case for hard-coded delivery fee estimation (#2042) Expose collection attributes from `Inspect` trait (#1914) `polkadot-parachain-primitives` should not depend on `frame-support`. (#1897) [testnet] Align testnet system parachain runtimes using `RelayTreasuryLocation` and `SystemParachains` in the same way (#2023) Sort the benchmarks before listing them (#2026) publish pallet-root-testing (#2017) Contracts: Add benchmarks to include files (#2022) Small optimisation to `--profile dev` wasm builds (#1851) basic-authorship: Improve time recording and logging (#2010) Application Crypto and BEEFY Support for paired (ECDSA,BLS) crypto (#1815) [ci] Run check-rust-feature-propagation in pr and master (#2012) Improve features dev-ex (#1831) Remove obsolete comment. (#2008) Refactor candidates test in paras_inherent (#2004) PVF: Add worker check during tests and benches (#1771) Bump actions/setup-node from 3.8.1 to 4.0.0 (#1997) polkadot: enable tikv-jemallocator/unprefixed_malloc_on_supported_platforms (#2002) Make `IdentityInfo` generic in `pallet-identity` (#1661) Ensure correct variant count in `Runtime[Hold/Freeze]Reason` (#1900) `CheckWeight`: Add more logging (#1996) ...

@koute

`wasm-builder` was adjusted to default to building wasm blobs in `release` mode even when cargo is in `debug` because `debug` wasm is too slow. A side effect of this was `.compact` and `.compact.compressed` getting built when the dev is running build in `debug`, adding ~5s to the build time of every wasm runtime. I think it's reasonable to assume if the dev is running `debug` build they want to optimise speed and do not care about the size of the wasm binary. Compacting a blob has negligible impact on its actual performance. In this PR, I adjusted the behavior of the wasm builder so it does not produce `.compact` or `.compact.compressed` wasm when the user is running in `debug`. The builder will continue to produce the bloaty wasm in release mode unless it is overriden with an env var. As suggested by @koute in review, also refactored the `maybe_compact_wasm_and_copy_blobs` into multiple funuctions, and renamed things to better support RISC-V in the future. --- There is no `T-runtime` label so @KiChjang told me to put `T1-FRAME` :) --------- Co-authored-by: Koute <koute@users.noreply.github.com>

@athei

This PR fixes a subtle bug in `wasm-builder` first introduced in #1851 (sorry, my bad! I should have caught this during review) where the status code of the `cargo` subprocess is not properly checked, which results in builds silently succeeding when they shouldn't (that is: if we successfully build a runtime blob, and then modify the code so that it won't compile, and recompile it again, then the build will succeed and silently use the *old* blob). cc @athei This is the bug you were seeing. [edit]Also fixes a similar PolkaVM-specific bug where I accidentally used the wrong comparison operator.[/edit]

@athei

This PR fixes a subtle bug in `wasm-builder` first introduced in paritytech#1851 (sorry, my bad! I should have caught this during review) where the status code of the `cargo` subprocess is not properly checked, which results in builds silently succeeding when they shouldn't (that is: if we successfully build a runtime blob, and then modify the code so that it won't compile, and recompile it again, then the build will succeed and silently use the *old* blob). cc @athei This is the bug you were seeing. [edit]Also fixes a similar PolkaVM-specific bug where I accidentally used the wrong comparison operator.[/edit]

@koute

`wasm-builder` was adjusted to default to building wasm blobs in `release` mode even when cargo is in `debug` because `debug` wasm is too slow. A side effect of this was `.compact` and `.compact.compressed` getting built when the dev is running build in `debug`, adding ~5s to the build time of every wasm runtime. I think it's reasonable to assume if the dev is running `debug` build they want to optimise speed and do not care about the size of the wasm binary. Compacting a blob has negligible impact on its actual performance. In this PR, I adjusted the behavior of the wasm builder so it does not produce `.compact` or `.compact.compressed` wasm when the user is running in `debug`. The builder will continue to produce the bloaty wasm in release mode unless it is overriden with an env var. As suggested by @koute in review, also refactored the `maybe_compact_wasm_and_copy_blobs` into multiple funuctions, and renamed things to better support RISC-V in the future. --- There is no `T-runtime` label so @KiChjang told me to put `T1-FRAME` :) --------- Co-authored-by: Koute <koute@users.noreply.github.com>

@athei

This PR fixes a subtle bug in `wasm-builder` first introduced in paritytech#1851 (sorry, my bad! I should have caught this during review) where the status code of the `cargo` subprocess is not properly checked, which results in builds silently succeeding when they shouldn't (that is: if we successfully build a runtime blob, and then modify the code so that it won't compile, and recompile it again, then the build will succeed and silently use the *old* blob). cc @athei This is the bug you were seeing. [edit]Also fixes a similar PolkaVM-specific bug where I accidentally used the wrong comparison operator.[/edit]

…aritytech#1851) * fix `HeadersToKeep` and `MaxBridgedAuthorities` in Millau benchmarks * typo * impl review suggestion

…1851) * fix `HeadersToKeep` and `MaxBridgedAuthorities` in Millau benchmarks * typo * impl review suggestion

…aritytech#1851) * fix `HeadersToKeep` and `MaxBridgedAuthorities` in Millau benchmarks * typo * impl review suggestion

liamaharon added 3 commits October 11, 2023 11:48

debug wasm build optimisation

22fae28

cleanup

0035c21

remove derive partialeq

1995cf2

liamaharon added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Oct 11, 2023

liamaharon requested a review from koute as a code owner October 11, 2023 12:52

franciscoaguirre approved these changes Oct 11, 2023

View reviewed changes

liamaharon added 2 commits October 22, 2023 18:56

Merge branch 'master' into liam-debug-wasm-optimisation

415a00e

Merge branch 'master' into liam-debug-wasm-optimisation

69b2ed0

koute reviewed Oct 23, 2023

View reviewed changes

substrate/utils/wasm-builder/src/wasm_project.rs Outdated Show resolved Hide resolved

substrate/utils/wasm-builder/src/wasm_project.rs Outdated Show resolved Hide resolved

substrate/utils/wasm-builder/src/wasm_project.rs Outdated Show resolved Hide resolved

liamaharon and others added 6 commits October 24, 2023 10:48

Update substrate/utils/wasm-builder/src/wasm_project.rs

edb604c

Co-authored-by: Koute <koute@users.noreply.github.com>

Merge branch 'master' into liam-debug-wasm-optimisation

772a102

rename build_wasm to build_blob

3cd37be

finish build config renaming

898cc42

koute refactor suggestion

be34e51

fix paths

96456eb

liamaharon requested a review from koute October 24, 2023 01:29

koute approved these changes Oct 24, 2023

View reviewed changes

substrate/utils/wasm-builder/src/wasm_project.rs Show resolved Hide resolved

substrate/utils/wasm-builder/src/wasm_project.rs Outdated Show resolved Hide resolved

liamaharon added 6 commits October 24, 2023 15:04

Merge branch 'master' into liam-debug-wasm-optimisation

6af92bf

use if instead of match

c1430e9

remove unnecesarry log

7f55d20

fix out name override

b7a21a0

remove dbg

d5af378

Merge branch 'master' into liam-debug-wasm-optimisation

151dec8

liamaharon merged commit ff3a3bc into master Oct 25, 2023
110 checks passed

liamaharon deleted the liam-debug-wasm-optimisation branch October 25, 2023 00:02

kiltbot mentioned this pull request Nov 7, 2023

[AUTOMATIC] Update Polkadot dependencies from 1.2.0 to 1.3.0 KILTprotocol/kilt-node#575

Open

ahmadkaouk mentioned this pull request Nov 16, 2023

Update polkadot-sdk from v.1.1.0 to v1.3.0 moonbeam-foundation/moonbeam#2565

Closed

librelois mentioned this pull request Dec 14, 2023

Compact and compressed wasm blob are not produced by default now moonbeam-foundation/moonbeam-runtime-overrides#83

Merged

koute mentioned this pull request Feb 22, 2024

Fix wasm-builder not exiting if compilation fails #3439

Merged

bkchr pushed a commit that referenced this pull request Apr 10, 2024

Fix HeadersToKeep and MaxBridgedAuthorities in Millau benchmarks (#…

cd12d14

…1851) * fix `HeadersToKeep` and `MaxBridgedAuthorities` in Millau benchmarks * typo * impl review suggestion

ukint-vs mentioned this pull request Apr 25, 2024

feat: Update substrate to v1.3.0 gear-tech/gear#3921

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Small optimisation to `--profile dev` wasm builds #1851

Small optimisation to `--profile dev` wasm builds #1851

liamaharon commented Oct 11, 2023 •

edited

Small optimisation to --profile dev wasm builds #1851

Small optimisation to --profile dev wasm builds #1851

Conversation

liamaharon commented Oct 11, 2023 • edited

Small optimisation to `--profile dev` wasm builds #1851

Small optimisation to `--profile dev` wasm builds #1851

liamaharon commented Oct 11, 2023 •

edited