Staking Miner #3141

kianenigma · 2021-05-31T14:05:49Z

List of substrate PRs that need to be merged before this is usable:

Companion for Decouple Staking and Election - Part 3: Signed Phase #2793 Decouple Staking and Election - Part 3: Signed Phase substrate#7910 (review)
https://github.com/paritytech/substrate/pull/8912/files and all of its child PRs
Update FallbackStrategy to Nothing #3088

paritytech/substrate#7910

…7910

…dot into kiz-staking-miner

runtime/polkadot/src/lib.rs

utils/staking-miner/Cargo.toml

utils/staking-miner/src/dry_run.rs

utils/staking-miner/src/main.rs

utils/staking-miner/src/dry_run.rs

utils/staking-miner/src/main.rs

utils/staking-miner/src/monitor.rs

utils/staking-miner/src/main.rs

…ing-miner

utils/staking-miner/src/main.rs

runtime/polkadot/src/constants.rs

runtime/polkadot/src/lib.rs

runtime/westend/src/lib.rs

emostov · 2021-07-01T18:25:38Z

utils/staking-miner/src/emergency_solution.rs

+			let encoded_ready = ready_solution.encode();
+			let encoded_support = ready_solution.supports.encode();
+			let mut solution_file = std::fs::File::create("solution.bin")?;
+			let mut supports_file = std::fs::File::create("solution.supports.bin")?;


i assume this works with no ./ or /full/path/to/file

emostov · 2021-07-01T18:26:51Z

utils/staking-miner/src/emergency_solution.rs

+			Ok(())
+		})
+	}
+}}}


Nothing to change for now, but in the future would nice to have cli options for

solution_file

supports_file

iters (min solution)

emostov · 2021-07-01T18:46:56Z

utils/staking-miner/src/main.rs

+macro_rules! any_runtime {
+	($($code:tt)*) => {
+		unsafe {
+			match $crate::RUNTIME {


Not something to change for this draft, but in the future it would be nice to not have to use this RUNTIME and instead figure out a way to explicitly pass it in as an arg. Maybe even there is some builder model we could come up with?

emostov · 2021-07-01T18:49:21Z

utils/staking-miner/src/monitor.rs

+
+		while let Ok(Some(now)) = subscription.next().await {
+			let hash = now.hash();
+			log::debug!(target: LOG_TARGET, "new event at #{:?} ({:?})", now.number, hash);


Suggested change

log::debug!(target: LOG_TARGET, "new event at #{:?} ({:?})", now.number, hash);

log::debug!(target: LOG_TARGET, "new subscription event at #{:?} ({:?})", now.number, hash);

just not to be confused with a frame Event

emostov · 2021-07-01T18:52:40Z

utils/staking-miner/src/monitor.rs

+			let (raw_solution, witness) = crate::mine_unchecked::<Runtime>(&mut ext, config.iterations, true)?;
+			log::info!(target: LOG_TARGET, "mined solution with {:?}", &raw_solution.score);
+
+			let nonce = crate::get_account_info::<Runtime>(&client, &signer.account, Some(hash))


would be nice if we could assume the nonce some way to speed up the critical path to solution submission. only thing i can think of is if we querried before this subscription loop, but that could probs run into other issues.

consider using accountNextIndex rpc as I think they takes into account pending txs in the queue

emostov

For a first draft it looks good to me.

main feedback

There should be some clear warnings around loss of funds from the given URI. UPDATE: I see good warning in the docs at the top or main.rs - whoever is writing release notes may want to hoist those docs.
With all the expects (especially the ones on RPC calls that are fallible even in ok conditions) we must rely on restarting the service. If we restart during the signed phase, the start up time is fairly important (we don't want to miss the entire signed phase and less important we don't want to get beaten by other miners). Given we are at least loading the EPM pallet storage I suspect this will take at min a few minutes. In the future it would be nice if we could store the chain state to file, and then check the file on restart (or even better not have to restart and keep it in memory :D)
make sure to revert changes to polkadots SessionsPerEra and EPOCH_DURATION_IN_SLOTS if they where not intentional
(this may be out of date feedback now)

Review limitations:

I did not review functions under #[allow(unused)]
My macro syntax understanding is primitive, so could easily have missed details within the macro impls

* master: (21 commits) cleanup stream polls (#3397) Staking Miner (#3141) Companion for Substrate#8953 (#3140) Bump version, specs & substrate in prep for v0.9.8 (#3387) Fix busy loops. (#3392) Minor refactor (#3386) add simnet tests (#3381) BEEFY: adjust gossip (#3372) Companion for #9193 (#3376) Companion for Decouple Staking and Election - Part 3: Signed Phase (#2793) Ensure that we fetch another collation if the first collation was invalid (#3362) Only send one collation per relay parent at a time to validators (#3360) disable approval-checking-grandpa on dev chain (#3364) Use associated constant for max (#3375) Use wasm-builder from git (#3354) Squashed 'bridges/' changes from b2099c5..23dda62 (#3369) Bump versions & spec_versions (#3368) Don't allow bids for a ParaId where there is an overlapping lease period (#3361) Companion for upgrade of transaction-payment to pallet macro (#3267) Do not allow any crowdloan contributions during the VRF period (#3346) ...

* master: (33 commits) Update all weights, add run_all_benches.sh script (#3400) Enable over-bridge-messaging in Rococo/Wococo runtime (#3377) paras.rs to FRAME V2 (#3403) Add XCM Tracing (#3353) Use MaxEncodedLen trait from new parity-scale-codec v2.2 (#3412) bump a bunch of deps in parity-common (#3402) Warn on low connectivity. (#3408) origin to frame v2 (#3405) Enable logging in the puppet worker (#3411) make it easier to dbg stalls (#3351) XCM `canonicalize` + `prepend_with` fix (#3269) cleanup stream polls (#3397) Staking Miner (#3141) Companion for Substrate#8953 (#3140) Bump version, specs & substrate in prep for v0.9.8 (#3387) Fix busy loops. (#3392) Minor refactor (#3386) add simnet tests (#3381) BEEFY: adjust gossip (#3372) Companion for #9193 (#3376) ...

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com> Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well.

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well. Co-authored-by: Bastian Köcher <git@kchr.de>

coriolinus and others added 7 commits April 1, 2021 09:32

Companion for Decouple Staking and Election - Part 3: Signed Phase

36502dc

paritytech/substrate#7910

Merge remote-tracking branch 'origin/master' into prgn-companion-for-…

1e509e9

…7910

Merge remote-tracking branch 'origin/master' into prgn-companion-for-…

9638fcf

…7910

Merge branch 'prgn-companion-for-7910' of github.com:paritytech/polka…

59646c2

…dot into kiz-staking-miner

Template added

271d351

Add stuff

7e21f24

First working version

61cf070

github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label May 31, 2021

kianenigma added 2 commits May 31, 2021 20:06

revamp rpc stuff

95b6ea0

remove all the stale js stuff

1709494

coriolinus reviewed Jun 1, 2021

View reviewed changes

kianenigma added 3 commits June 3, 2021 10:50

Major updates, closer to completion.

0c10c66

removing my generic WIP for tidyness.

a4aa730

add --at

72db361

coriolinus reviewed Jun 9, 2021

View reviewed changes

utils/staking-miner/src/main.rs Show resolved Hide resolved

emostov self-requested a review June 9, 2021 17:10

kianenigma added 8 commits June 13, 2021 07:56

ass measure snapshot

dc1a623

some new little debugging thigs

5fa5a3f

pre-merge

138e8b9

Merge branch 'master' of github.com:paritytech/polkadot into kiz-stak…

c3cfe94

…ing-miner

Master.into()

863ea3a

Fix

3980e85

revert dep update

d1ba0cb

update to master

fe76cee

kianenigma marked this pull request as ready for review June 30, 2021 18:58

github-actions bot added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jun 30, 2021

kianenigma added B0-silent Changes should not be mentioned in any release notes C3-medium PR touches the given topic and has a medium impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Jun 30, 2021