Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 #2202

nbaztec · 2023-04-01T12:13:30Z

What does it do?

update to v0.9.40
introduces warp-sync

Important Changes

Substrate

Randomness Collective Flip Rename pallet-random-collective-flip to Insecure... paritytech/substrate#13301
- renames pallet-randomness-collective-flip to pallet-insecure-randomness-collective-flip to state the fact that the pallet should not be used for production use-cases.
Grandpa Offences report system rework paritytech/substrate#13425
- Moves sc-finality-grandpa to sc-consensus-grandpa
XCM [XCM] Multiple FungiblesAdapters support + WeightTrader::buy_weight more accurate error paritytech/polkadot#6739
- MatchError::AssetNotFound > MatchError::AssetNotHandled
Collective SetMembers configurable origin paritytech/substrate#13159
- SetMembersOrigin new type
Democracy Introduce EnsureOrigin to democracy.propose paritytech/substrate#13410
- adds SubmitOrigin
Channels Make unbounded channels size warning exact (part 1) paritytech/substrate#13490
- replace futures-channel with async-channel
KeyIterators Rework storage iterators paritytech/substrate#13284, Fix the storage_size/state_getStorageSize RPC call paritytech/substrate#13154
- Refactors storage_keys, storage_pairs, child_storage_keys.
Move code from sc-network-common back to sc-network paritytech/substrate#13592
- Moves code from sc-network-common is moved to sc-network
Extract syncing protocol from sc-network paritytech/substrate#12828
- extracts the syncing protocol from sc-network

Cumulus

relay-chain-interface: Do not depend on polkadot-service paritytech/cumulus#2287
- Changes RelayChainError::Application to RelayChainError::ServiceError (affects nimbus)

Frontier

pallet-ethereum feat: provide a parameter for the runtime config to choose what to include in the PostLog polkadot-evm/frontier#1014
- provides runtime config to choose what to include in the PostLog.

⚠️ Breaking Changes ⚠️

SetMembersOrigin is now configured to use also the GeneralAdmin track

crystalin · 2023-04-11T08:40:16Z

It seems to fail with cargo build --release --features=try-runtime also

crystalin · 2023-04-11T09:02:35Z

I also tried ./target/release/moonbeam --base-path /tmp/warp-test --sync warp --blocks-pruning 256 --state-pruning 256 -- --sync warp --blocks-pruning 256 --state-pruning 256 and get spamming error:

2023-04-11 11:01:50 [Relaychain] if watch returned an error: rtnetlink socket closed
2023-04-11 11:01:50 [Relaychain] failed to read from netlink socket: Custom { kind: Other, error: "A Tokio 1.x context was found, but it is being shutdown." }
2023-04-11 11:01:50 [Relaychain] netlink socket stream shut down

…upgrade-v0.9.40

nbaztec · 2023-04-19T10:14:02Z

build works now, but warp-sync still fails with:

$ ./target/release/moonbeam --base-path /tmp/warp-test --sync warp --blocks-pruning 256 --state-pruning 256  -- --sync warp --blocks-pruning 256 --state-pruning 256
2023-04-18 18:49:16 Moonbeam Parachain Collator    
2023-04-18 18:49:16 ✌️  version 0.31.0-02b81f3ff04    
2023-04-18 18:49:16 ❤️  by PureStake, 2019-2023    
2023-04-18 18:49:16 📋 Chain specification: Moonbeam    
2023-04-18 18:49:16 🏷  Node name: absurd-rail-9979    
2023-04-18 18:49:16 👤 Role: FULL    
2023-04-18 18:49:16 💾 Database: RocksDb at /tmp/warp-test/chains/moonbeam/db/full    
2023-04-18 18:49:16 ⛓  Native runtime: moonbeam-2300 (moonbeam-0.tx2.au3)    
2023-04-18 18:49:17 Parachain id: Id(2004)    
2023-04-18 18:49:17 Parachain Account: 5Ec4AhPVjsshXjh8ynp6MwaJTJBnen3pkHiiyDhHfie5VWkN    
2023-04-18 18:49:17 Parachain genesis state: 0x00000000000000000000000000000000000000000000000000000000000000000074e54bafcf1ecdce51867078fa16f2902751d11473628ac19e89a5be20df326203170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400    
2023-04-18 18:49:17 Is collating: no    
2023-04-18 18:49:18 [🌗] 🔨 Initializing Genesis block/state (state: 0x74e5…3262, header-hash: 0xfe58…b76d)    
2023-04-18 18:49:18 [🌗] 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.    
Error: Service(Client(Execution(Other("Exported method GrandpaApi_grandpa_authorities is not found"))))

node/service/src/lib.rs

node/cli/src/command.rs

crystalin · 2023-05-22T20:15:02Z

@girazoki the test for XCM fails
node ./node_modules/.bin/mocha -r ts-node/register tests/test-xcm/test-mock-dmp-queue.ts

[
  'treasury.UpdatedInactive.[875624120372174,1382629026773089]',
  'system.ExtrinsicSuccess.[{"weight":{"refTime":259098000,"proofSize":1006},"class":"Mandatory","paysFee":"Yes"}]',
  'parachainSystem.DownwardMessagesReceived.[20]',
  'balances.Withdraw.["0x506172656E740000000000000000000000000000",1000000000000000]',
  'balances.Deposit.["0x6d6F646c70632f74727372790000000000000000",122864400000000]',
  'treasury.Deposit.[122864400000000]',
  'polkadotXcm.AssetsTrapped.["0x5346907f0d1a1614c52b7bc79d98161f7dc37c415691a8229d4b1165024f1c52",{"parents":1,"interior":{"here":null}},{"v3":[{"id":{"concrete":{"parents":0,"interior":{"x1":{"palletInstance":3}}}},"fun":{"fungible":385678000000000}}]}]',
  'dmpQueue.ExecutedDownward.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",{"complete":{"refTime":12286440000,"proofSize":826650}}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",9,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",10,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",11,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",12,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",13,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",14,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",15,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",16,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.OverweightEnqueued.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0",17,{"refTime":12286440000,"proofSize":826650}]',
  'dmpQueue.MaxMessagesExhausted.["0xdfac9648a6d2babcedf34d0ae2c120771fdbb6247eccd38d20de1d8cf81cc9c0"]'
]

The test is supposed to be computing the weight but I don't understand what it gets wrong

…ot have pov

This reverts commit 882b85e.

github-actions · 2023-05-23T16:38:04Z

Coverage generated "Thu May 25 07:47:38 UTC 2023":
https://s3.amazonaws.com/moonbeam-coverage/pulls/2202/html/index.html

Master coverage: 72.61%
Pull coverage: 70.88%

precompiles/utils/macro/src/precompile/expand.rs

notlesh

LGTM. I'm concerned about some of the new proof_size benchmark values, do you think they might reduce block fullness if they are overestimates? (If they're underestimates, they're still better than 0...) See my other comment for more context.

notlesh · 2023-05-24T21:48:57Z

pallets/parachain-staking/src/weights.rs

+		//  Measured:  `2479 + x * (79 ±0) + y * (38 ±0)`
+		//  Estimated: `32988 + x * (405 ±0) + y * (195 ±0)`
+		// Minimum execution time: 94_976_000 picoseconds.
+		Weight::from_parts(83_764_148, 32988)


This storage_proof size is probably excessively high because it depends on at least one Vec in storage... eww.

Notice that the generated code here has _ separators for time but not proof size. I'm guessing Parity doesn't have many benchmarks where proof_size > 999...

notlesh · 2023-05-24T22:05:31Z

tests/util/xcm.ts

@@ -290,6 +290,17 @@ export class XcmFragment {
    return this;
  }

+  // Add one or more `BuyExecution` instruction


Suggested change

// Add one or more `BuyExecution` instruction

// Add one or more `RefundSurplus` instruction(s)

notlesh · 2023-05-24T22:06:52Z

tests/tests/test-staking/test-staking-locks.ts

-            .joinCandidates(MIN_GLMR_STAKING, maxDelegationsPerDelegator)
-            .signAsync(randomCandidate)
+    // We split the candidates since they won't fit in a single block
+    for (const randomCandidatesChunk of chunk(randomCandidates, 50)) {


This test always breaks, hopefully this alleviates that! 👍

crystalin · 2023-05-25T12:08:17Z

pallets/erc20-xcm-bridge/src/lib.rs

 				T::Erc20TransferGasLimit::get().saturating_mul(T::WeightPerGas::get().ref_time()),
+				0,


@librelois is 0 good here ?

Good catch, it's not good, I will fix that on a follow up PR

outdated/resolved

* update to v0.9.40 * update benchmarking weight template * fix build * make test compile * Includes page heap fixes * compile runtime-benchmarks * make warp sync work * toml sort * fix editorconfig * use new substrate version * fix warp sync * sort * fix --dev * remove duplicate SetMembersOrigin * toml-sort * remove kitchensink-runtime * fix builkd * use new weights * set manual weights for xcm fungible * use Weight::from_parts * use 0 pov_size for ref_time weight * update nimbus * exclude generated weight files from editorconfig * fmt * fmt * fix rust tests * fix import * fix tests * use Weight part pov_size to 0 * make dalek test work * fix transfer tests * use BoundedVec for auto compound delegations * fix modexp test * fix modexp test * fix tests * fix weight tests * fix staking tests via chunking * fix modexp test * fix lint and test * fix rust weight tests * fix partial ts tests * temp fix for xcm v2 * Fixes weight until benchmarking is fixed * set manual weight, fix ts tests * Adds temp hack for xcm tests * Use RefundSurplus as the no-op for saturating the queue, which does not have pov * Update evm to 0.39 * Revert "Update evm to 0.39" This reverts commit 882b85e. * upgrade polkadot for better support of xcm v2 * prettier * prettier * Revert temp fix for XCM weight * upgrade polkadot fork * Fixing hrmp-mock tests * clean up * prettier --------- Co-authored-by: crystalin <alan.sapede@gmail.com> Co-authored-by: librelois <c@elo.tf> Co-authored-by: girazoki <gorka.irazoki@gmail.com> Co-authored-by: tgmichel <telmo@purestake.com>

nbaztec added 3 commits April 1, 2023 14:13

update to v0.9.40

6d394d0

update benchmarking weight template

d26b671

fix build

2821987

nbaztec added B7-runtimenoteworthy Changes should be noted in any runtime-upgrade release notes D9-needsaudit👮 PR contains changes to fund-managing logic that should be properly reviewed and externally audited breaking Needs to be mentioned in breaking changes labels Apr 3, 2023

nbaztec and others added 2 commits April 3, 2023 15:13

make test compile

1ed2a6d

Includes page heap fixes

fb2df2a

nbaztec added 4 commits April 18, 2023 15:18

compile runtime-benchmarks

c9c9faf

Merge branch 'upgrade-v0.9.40' of github.com:PureStake/moonbeam into …

02b81f3

…upgrade-v0.9.40

make warp sync work

2882824

merge master

7fd8681

nbaztec added 3 commits April 19, 2023 12:16

toml sort

a723757

fix editorconfig

471a4d2

use new substrate version

e10550f

bkchr reviewed Apr 20, 2023

View reviewed changes

node/service/src/lib.rs Outdated Show resolved Hide resolved

bkchr mentioned this pull request Apr 20, 2023

Unable to use warp-sync on parachain paritytech/substrate#13963

Closed

2 tasks

bkchr reviewed Apr 21, 2023

View reviewed changes

node/service/src/lib.rs Outdated Show resolved Hide resolved

nbaztec added 3 commits April 21, 2023 11:58

fix warp sync

0fb006c

merge conflicts

7d58b9d

sort

d76bc6b

nbaztec marked this pull request as ready for review April 24, 2023 10:08

nbaztec added 3 commits April 25, 2023 12:30

fix --dev

a4b0848

remove duplicate SetMembersOrigin

0e45f47

toml-sort

56ae6f8

bkchr reviewed Apr 26, 2023

View reviewed changes

node/cli/src/command.rs Outdated Show resolved Hide resolved

remove kitchensink-runtime

0ccee4a

crystalin and others added 2 commits May 16, 2023 14:28

Fixes weight until benchmarking is fixed

df27699

set manual weight, fix ts tests

7ef16e7

AurevoirXavier mentioned this pull request May 22, 2023

Replace the forked moonbeam deps darwinia-network/darwinia#1152

Closed

crystalin and others added 9 commits May 22, 2023 22:38

Adds temp hack for xcm tests

6669928

Use RefundSurplus as the no-op for saturating the queue, which does n…

0d1f026

…ot have pov

Update evm to 0.39

882b85e

Revert "Update evm to 0.39"

389f757

This reverts commit 882b85e.

upgrade polkadot for better support of xcm v2

d831644

prettier

63c005a

prettier

ea906b9

Revert temp fix for XCM weight

f7d6b37

upgrade polkadot fork

f54cc1b

girazoki added 2 commits May 24, 2023 10:43

Fixing hrmp-mock tests

6c044fc

clean up

1608234

nanocryk previously requested changes May 24, 2023

View reviewed changes

precompiles/utils/macro/src/precompile/expand.rs Outdated Show resolved Hide resolved

notlesh reviewed May 24, 2023

View reviewed changes

librelois added 2 commits May 25, 2023 08:41

Merge branch 'master' into upgrade-v0.9.40

a874fc4

prettier

0ca0b68

librelois approved these changes May 25, 2023

View reviewed changes

crystalin approved these changes May 25, 2023

View reviewed changes

crystalin merged commit 62ddcf2 into master May 25, 2023
26 checks passed

crystalin deleted the upgrade-v0.9.40 branch May 25, 2023 12:28

wilwade mentioned this pull request Jun 6, 2023

Enable Warp Sync frequency-chain/frequency#1585

Closed

crystalin changed the title ~~update to v0.9.40~~ Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 Jun 9, 2023

notlesh added D1-audited👍 PR contains changes to fund-managing logic that has been properly reviewed and externally audited and removed D9-needsaudit👮 PR contains changes to fund-managing logic that should be properly reviewed and externally audited labels Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 #2202

Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 #2202

nbaztec commented Apr 1, 2023 •

edited by crystalin

crystalin commented Apr 11, 2023

crystalin commented Apr 11, 2023

nbaztec commented Apr 19, 2023

crystalin commented May 22, 2023

github-actions bot commented May 23, 2023 •

edited

notlesh left a comment

notlesh May 24, 2023

notlesh May 24, 2023

notlesh May 24, 2023

notlesh May 24, 2023

crystalin May 25, 2023

librelois May 25, 2023

	// Add one or more `BuyExecution` instruction
	// Add one or more `RefundSurplus` instruction(s)

		T::Erc20TransferGasLimit::get().saturating_mul(T::WeightPerGas::get().ref_time()),
		0,

Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 #2202

Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 #2202

Conversation

nbaztec commented Apr 1, 2023 • edited by crystalin

What does it do?

Important Changes

Substrate

Cumulus

Frontier

⚠️ Breaking Changes ⚠️

crystalin commented Apr 11, 2023

crystalin commented Apr 11, 2023

nbaztec commented Apr 19, 2023

crystalin commented May 22, 2023

github-actions bot commented May 23, 2023 • edited

notlesh left a comment

Choose a reason for hiding this comment

notlesh May 24, 2023

Choose a reason for hiding this comment

notlesh May 24, 2023

Choose a reason for hiding this comment

notlesh May 24, 2023

Choose a reason for hiding this comment

notlesh May 24, 2023

Choose a reason for hiding this comment

crystalin May 25, 2023

Choose a reason for hiding this comment

librelois May 25, 2023

Choose a reason for hiding this comment

nbaztec commented Apr 1, 2023 •

edited by crystalin

github-actions bot commented May 23, 2023 •

edited