Staking v4: soft auction list selection #4083

mariusmihaic · 2022-05-17T10:45:39Z

Implemented soft(dynamic) auction selection for nodes in auction list. "Core" implementation can be found in calcSoftAuctionNodesConfig func, which implements the algorithm proposed here (see Pseudo code of topUp calculation)

What's new

Added SoftAuctionConfig config for soft auction selection
Extended StakingDataProvider to provide GetNumStakedNodes & GetTotalTopUp of an owner. Note that it is required that these information are pre-filled (which happens for all nodes in ProcessSystemSC) before calling SelectNodesFromAuctionList
SelectNodesFromAuctionList now expects to receive all unqualified owners, of whom auction nodes will not participate at selection. These unqualified owners are the ones which do not have enough stake for their nodes, therefore one or more of their nodes will be removed at the end of the epoch.
Each node in auction will have its ownerData computed and filled(active nodes, auction nodes, topUp per node, etc...) before applying the dynamic selection. After soft auction computation is done, each owner's auction nodes will be sorted (BY blsKey) and qualified ones will be marked as selected. If two/more nodes, which belong to different owners, have the same qualifiedTopUpPerNode, sorting will be done based on blsKey XOR randomness. See TestSystemSCProcessor_ProcessSystemSmartContractStakingV4Enabled test which contains an example with graphic tables

Bugfixes

Fixed bug in epochStart/metachain/systemSCs.go when preparing staking data (L134). It was done for all nodes instead of eligible nodes only, causing rewards to be distributed to non-eligible nodes.

…implement-soft-auction # Conflicts: # epochStart/errors.go # epochStart/metachain/auctionListSelector.go # epochStart/metachain/auctionListSelector_test.go

codecov-commenter · 2022-05-23T13:07:56Z

Codecov Report

Merging #4083 (2623fb7) into feat/liquid-staking (363865a) will increase coverage by 0.61%.
The diff coverage is 89.16%.

@@                   Coverage Diff                   @@
##           feat/liquid-staking    #4083      +/-   ##
=======================================================
+ Coverage                75.38%   75.99%   +0.61%     
=======================================================
  Files                      622      658      +36     
  Lines                    83275    86766    +3491     
=======================================================
+ Hits                     62774    65936    +3162     
- Misses                   15776    15988     +212     
- Partials                  4725     4842     +117

Impacted Files	Coverage Δ
dataRetriever/interface.go	`12.12% <ø> (ø)`
epochStart/bootstrap/baseStorageHandler.go	`55.55% <ø> (ø)`
epochStart/metachain/common.go	`0.00% <0.00%> (ø)`
factory/bootstrapComponentsHandler.go	`55.55% <0.00%> (ø)`
factory/consensusComponentsHandler.go	`60.56% <0.00%> (+4.71%)`	⬆️
factory/cryptoComponentsHandler.go	`66.21% <0.00%> (ø)`
factory/dataComponentsHandler.go	`71.08% <0.00%> (ø)`
factory/heartbeatComponentsHandler.go	`64.17% <0.00%> (ø)`
factory/stateComponentsHandler.go	`72.04% <0.00%> (ø)`
factory/statusComponentsHandler.go	`16.45% <0.00%> (ø)`
... and 145 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47ee099...2623fb7. Read the comment docs.

# Conflicts: # cmd/node/config/config.toml # config/config.go # testscommon/generalConfig.go

iulianpascalau

partial review, mostly on calcSoftAuctionNodesConfig function

iulianpascalau · 2022-05-27T07:42:13Z

epochStart/metachain/auctionListSelector.go

+	sw.Start("auctionListSelector.sortAuctionList")
+	defer func() {
+		sw.Stop("auctionListSelector.sortAuctionList")
+		log.Debug("time measurements", sw.GetMeasurements()...)


iulianpascalau · 2022-05-27T07:43:02Z

epochStart/metachain/auctionListSelector.go

-
-		ret[string(pubKey)] = topUp
+// TODO: Move this in elrond-go-core
+func safeSub(a, b uint32) (uint32, error) {


we have this in core already I think.

Not for uint32, only for uint64 and I didn't want to cast the values

iulianpascalau · 2022-05-27T07:55:53Z

epochStart/metachain/auctionListSelector.go

+		numNodesQualifyingForTopUp := int64(0)
+		previousConfig = copyOwnersData(ownersData)
+
+		for ownerPubKey, owner := range ownersData {


suggestion to extract L335-L357 in a separate function that will output the required map & numNodesQualifyingForTopUp value

Extracted separate function

sasurobert · 2022-05-30T09:03:49Z

epochStart/metachain/stakingDataProvider.go

+
+// GetTotalTopUp returns owner's total top up
+func (sdp *stakingDataProvider) GetTotalTopUp(owner []byte) (*big.Int, error) {
+	ownerInfo, ok := sdp.cache[string(owner)]


we have some mutexes for other gets. add here as well

+1 on all gets from the cache

This is now obsolete, since this func is removed in future refactor PR

sasurobert · 2022-05-30T09:10:48Z

epochStart/metachain/auctionListSorting.go

+	return selectedFromAuction[:numAvailableSlots]
+}
+
+func getPubKeyLen(ownersData map[string]*ownerData) int {


what is this ? you do not need to iterate and return the lenght of data like this.

This does not iterate over all values; it stops at the first element; I need it to find the blsKeyLen so we can normalize randomness

sasurobert · 2022-05-30T09:16:45Z

epochStart/metachain/auctionListSelector.go

-
-		ret[string(pubKey)] = topUp
+// TODO: Move this in elrond-go-core
+func safeSub(a, b uint32) (uint32, error) {


we have this in core already I think.

sasurobert · 2022-05-30T09:44:02Z

epochStart/metachain/auctionListSelector.go

+func (als *auctionListSelector) getAuctionDataAndNumOfValidators(
+	validatorsInfoMap state.ShardValidatorsInfoMapHandler,
+	unqualifiedOwners map[string]struct{},
+) (map[string]*ownerData, uint32, uint32, error) {


make a structure please instead of 3 returns and the error. So one struct and the error.

This is now obsolete, since this func is removed in future refactor PR

sasurobert · 2022-05-30T09:46:57Z

epochStart/metachain/auctionListSelector.go

 	numOfValidators := uint32(0)
+	numOfNodesInAuction := uint32(0)

 	for _, validator := range validatorsInfoMap.GetAllValidatorsInfo() {


I think this is like double iteration - you already have the owner data in the stakingProvider component. Everything is there. You have to iterate per owner. Not per key. There are a lot of useless checks per key.

Yes, refactored all code in a separate PR: #4155

sasurobert · 2022-05-30T09:48:36Z

epochStart/metachain/auctionListSelector.go

+				continue
+			}
+
+			err = als.addOwnerData(owner, validator, ownersData)


I think you should start from data from the stakingProvider component and add the necessary information. You need to know the keys which are in the auction list for this owner and the total topup and other global info about the owner.

Yes, refactored all code in a separate PR: #4155

sasurobert · 2022-05-30T09:49:35Z

epochStart/metachain/auctionListSelector.go

+		numNodesQualifyingForTopUp := int64(0)
+		previousConfig = copyOwnersData(ownersData)
+
+		for ownerPubKey, owner := range ownersData {


cmd/node/config/config.toml

epochStart/metachain/systemSCs.go

raduchis · 2022-05-31T13:52:56Z

epochStart/metachain/auctionListSelector.go

+func (als *auctionListSelector) getAuctionDataAndNumOfValidators(
+	validatorsInfoMap state.ShardValidatorsInfoMapHandler,
+	unqualifiedOwners map[string]struct{},
+) (map[string]*ownerData, uint32, uint32, error) {


raduchis · 2022-05-31T13:55:38Z

epochStart/metachain/auctionListSelector.go

+		numNodesQualifyingForTopUp := int64(0)
+		previousConfig = copyOwnersData(ownersData)
+
+		for ownerPubKey, owner := range ownersData {


epochStart/metachain/auctionListSorting.go

epochStart/metachain/auctionListDisplayer.go

raduchis · 2022-05-31T14:24:30Z

epochStart/metachain/stakingDataProvider.go

+
+// GetTotalTopUp returns owner's total top up
+func (sdp *stakingDataProvider) GetTotalTopUp(owner []byte) (*big.Int, error) {
+	ownerInfo, ok := sdp.cache[string(owner)]


+1 on all gets from the cache

FEAT: First ugly version

8b4d1b8

mariusmihaic changed the title ~~FEAT: First ugly version~~ Staking v4: soft auction list selection May 17, 2022

mariusmihaic self-assigned this May 17, 2022

mariusmihaic added the DO NOT MERGE label May 17, 2022

mariusmihaic added 16 commits May 17, 2022 15:22

FEAT: First ugly working version

2f9c1c8

FEAT: Intermediary code

e7f6b9c

FEAT: Stable code

0ab80fc

FEAT: Do not add unqualified nodes in auction

e9ca4d3

CLN: Quick fix broken test

2c41f17

FIX: not selecting unqualified nodes for auction

dcf9f5b

CLN: Start refactor

f06c188

CLN: Refactor 2

a5659dc

CLN: Refactor 3

7e22f59

CLN: Refactor 4

900ed74

CLN: Refactor 5

c9f2fb0

Merge branch 'EN-12187-auction-list-selection-subcomp' into EN-12196-…

4f21dc5

…implement-soft-auction # Conflicts: # epochStart/errors.go # epochStart/metachain/auctionListSelector.go # epochStart/metachain/auctionListSelector_test.go

FIX: After merges

5a363a0

FIX: After merges 2

c3217c1

CLN: Refactor 5

b932f59

CLN: Refactor 6

b3b9129

Base automatically changed from EN-12187-auction-list-selection-subcomp to feat/liquid-staking May 23, 2022 11:48

CLN: Refactor 7

fd6898f

mariusmihaic added 7 commits May 24, 2022 11:56

FEAT: First test for calcSoftAuctionNodesConfig + bugfixes

b162246

CLN: Test

ba05416

FEAT: Add edge case tests for calcSoftAuctionNodesConfig

8d324c9

FEAT: > 99% code coverage

43a8338

FEAT: Add SoftAuctionConfig and integrate it

2a760b9

FEAT: Add getAuctionConfig test + split test files

09d3efc

FIX: Broken tests

8dd0ee3

mariusmihaic added 3 commits May 26, 2022 14:26

FIX: General fixes 1

64ef325

FIX: General fixes 2

85d08d9

Merge branch 'feat/liquid-staking' into EN-12196-implement-soft-auction

75e0b5a

mariusmihaic assigned mariusmihaic and unassigned mariusmihaic May 26, 2022

mariusmihaic removed the DO NOT MERGE label May 26, 2022

mariusmihaic marked this pull request as ready for review May 26, 2022 13:10

FIX: General fixes 3

a05cdd3

sasurobert self-requested a review May 27, 2022 06:44

Merge branch 'feat/liquid-staking' into EN-12196-implement-soft-auction

2623fb7

# Conflicts: # cmd/node/config/config.toml # config/config.go # testscommon/generalConfig.go

iulianpascalau reviewed May 27, 2022

View reviewed changes

raduchis self-requested a review May 30, 2022 06:51

sasurobert requested changes May 30, 2022

View reviewed changes

Merge branch 'feat/liquid-staking' into EN-12196-implement-soft-auction

0ea0111

raduchis reviewed May 31, 2022

View reviewed changes

FIX: Review findings

5e24f07

raduchis approved these changes Jun 3, 2022

View reviewed changes

sasurobert approved these changes Jun 7, 2022

View reviewed changes

mariusmihaic merged commit b546bdb into feat/liquid-staking Jun 8, 2022

mariusmihaic deleted the EN-12196-implement-soft-auction branch June 8, 2022 07:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Staking v4: soft auction list selection #4083

Staking v4: soft auction list selection #4083

mariusmihaic commented May 17, 2022 •

edited

codecov-commenter commented May 23, 2022 •

edited

iulianpascalau left a comment

iulianpascalau May 27, 2022

mariusmihaic Jun 3, 2022

iulianpascalau May 27, 2022

sasurobert May 30, 2022

mariusmihaic Jun 3, 2022

iulianpascalau May 27, 2022

sasurobert May 30, 2022

raduchis May 31, 2022

mariusmihaic Jun 3, 2022

sasurobert May 30, 2022

raduchis May 31, 2022

mariusmihaic Jun 3, 2022

sasurobert May 30, 2022

mariusmihaic Jun 3, 2022

sasurobert May 30, 2022

sasurobert May 30, 2022

raduchis May 31, 2022

mariusmihaic Jun 3, 2022

sasurobert May 30, 2022

mariusmihaic Jun 3, 2022

sasurobert May 30, 2022

mariusmihaic Jun 3, 2022 •

edited

sasurobert May 30, 2022

raduchis May 31, 2022

raduchis May 31, 2022

raduchis May 31, 2022

Staking v4: soft auction list selection #4083

Staking v4: soft auction list selection #4083

Conversation

mariusmihaic commented May 17, 2022 • edited

What's new

Bugfixes

codecov-commenter commented May 23, 2022 • edited

Codecov Report

iulianpascalau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariusmihaic Jun 3, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariusmihaic commented May 17, 2022 •

edited

codecov-commenter commented May 23, 2022 •

edited

mariusmihaic Jun 3, 2022 •

edited