MX-14120: repopulate tokens supplies #5204

bogdan-rosianu · 2023-04-27T12:27:53Z

Reasoning behind the pull request

previously, tokens supplies could have only been re-computed via an import-db which could have taken a few days, so we needed a faster mechanism for computing the tokens supplies

Proposed changes

add a CLI flag --repopulate-tokens-supplies that will clear the existing supplies database, will iterate over the trie and recompute the supplies and then save them to the db
NOTE: this feature does not compute the minted and burned values. They will be later indexed via the indexer

Testing procedure

run an observing squad on testnet/devnet/mainnet by starting the nodes with the --repopulate-tokens-supplies and then compare the supplies obtained by comparing the proxy instances via the /network/esdt/supply/<token> endpoint

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

was the PR targeted to the correct branch?
if this is a larger feature that probably needs more than one PR, is there a feat branch created?
if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

…ies' into MX-14120-repopulate-tokens-supplies

codecov · 2023-04-27T12:42:09Z

Codecov Report

Patch coverage: 80.95% and project coverage change: +0.10 🎉

Comparison is base (ffd2cb0) 79.59% compared to head (b5c34c8) 79.70%.

❗ Current head b5c34c8 differs from pull request most recent head 7029a48. Consider uploading reports for the commit 7029a48 to get more accurate results

Additional details and impacted files

@@              Coverage Diff              @@
##           rc/v1.6.0    #5204      +/-   ##
=============================================
+ Coverage      79.59%   79.70%   +0.10%     
=============================================
  Files            680      682       +2     
  Lines          88012    88183     +171     
=============================================
+ Hits           70053    70283     +230     
+ Misses         12837    12780      -57     
+ Partials        5122     5120       -2

Impacted Files	Coverage Δ
facade/nodeFacade.go	`97.25% <ø> (+18.77%)`	⬆️
process/sync/baseSync.go	`64.91% <9.52%> (-1.71%)`	⬇️
factory/consensus/consensusComponents.go	`96.24% <66.66%> (-0.39%)`	⬇️
process/sync/shardblock.go	`67.28% <66.66%> (-0.18%)`	⬇️
storage/factory/storageServiceFactory.go	`89.48% <68.00%> (-1.86%)`	⬇️
factory/processing/processComponents.go	`83.01% <83.33%> (-0.03%)`	⬇️
...cess/sync/trieIterators/tokensSuppliesProcessor.go	`89.87% <89.87%> (ø)`
epochStart/bootstrap/metaStorageHandler.go	`73.97% <100.00%> (+0.17%)`	⬆️
epochStart/bootstrap/process.go	`84.80% <100.00%> (+0.01%)`	⬆️
epochStart/bootstrap/shardStorageHandler.go	`88.00% <100.00%> (+0.02%)`	⬆️
... and 7 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

…ies' into MX-14120-repopulate-tokens-supplies

iulianpascalau · 2023-05-02T14:34:22Z

go.mod

@@ -13,7 +13,7 @@ require (
 	github.com/google/gops v0.3.18
 	github.com/gorilla/websocket v1.5.0
 	github.com/mitchellh/mapstructure v1.5.0
-	github.com/multiversx/mx-chain-core-go v1.2.1
+	github.com/multiversx/mx-chain-core-go v1.2.2-0.20230428142157-76e19ecd04ac


proper release after testing

iulianpascalau · 2023-05-02T14:37:16Z

node/nodeRunner.go

@@ -1227,7 +1228,7 @@ func (nr *nodeRunner) CreateManagedProcessComponents(
 		ImportStartHandler:     importStartHandler,
 		WorkingDir:             configs.FlagsConfig.WorkingDir,
 		HistoryRepo:            historyRepository,
-		SnapshotsEnabled:       configs.FlagsConfig.SnapshotsEnabled,
+		FlagsConfig:            *configs.FlagsConfig,


why not pass config by value? I consider this a safer way to pass arguments so no overwriting mistakes will have effects upstream. Also, WorkingDir also uses the configs.FlagsConfig.WorkingDir

it is already passed by value. updated the other fields

process/sync/shardblock.go

iulianpascalau · 2023-05-02T14:43:20Z

process/sync/trieIterators/tokensSuppliesComputer.go

+		suffix := append(userLeaf.Key(), userAccount.AddressBytes()...)
+		value, errVal := userLeaf.ValueWithoutSuffix(suffix)
+		if errVal != nil {
+			log.Warn("cannot get value without suffix", "error", errVal, "key", userLeaf.Key())


log & return error?

removed the log and wrapped the error

process/sync/baseSync.go

process/sync/trieIterators/tokensSuppliesComputer.go

iulianpascalau · 2023-05-03T06:49:51Z

process/sync/trieIterators/tokensSuppliesComputer.go

+			return err
+		}
+
+		tokenName := string(tokenKey)[lenESDTPrefix:]


should we test here that the tokenKey contains the ESDTPrefix?

no need, since we have this at the beginning of the for loop:

if !bytes.HasPrefix(userLeaf.Key(), esdtPrefix) { continue }

and also this: lenESDTPrefix := len(esdtPrefix)

process/sync/trieIterators/tokensSuppliesComputer.go

process/sync/trieIterators/tokensSuppliesComputer_test.go

process/sync/trieIterators/trieAccountsIterator.go

iulianpascalau · 2023-05-04T13:26:53Z

factory/consensus/consensusComponents.go

@@ -136,7 +136,10 @@ func (ccf *consensusComponentsFactory) Create() (*consensusComponents, error) {
 		return nil, err
 	}

-	cc.bootstrapper.StartSyncingBlocks()
+	err = cc.bootstrapper.StartSyncingBlocks()


iulianpascalau · 2023-05-04T13:30:50Z

factory/processing/processComponents.go

@@ -173,7 +171,6 @@ type processComponentsFactory struct {
 	txLogsProcessor        process.TransactionLogProcessor
 	version                string
 	importStartHandler     update.ImportStartHandler
-	workingDir             string


we removed the workingDir but not the version. The code will compile with an empty version string as you delete L221. Let's also remove this field and use the provided version instead.

very good catch. fixed

iulianpascalau · 2023-05-04T13:31:07Z

node/nodeRunner.go

@@ -1227,7 +1228,7 @@ func (nr *nodeRunner) CreateManagedProcessComponents(
 		ImportStartHandler:     importStartHandler,
 		WorkingDir:             configs.FlagsConfig.WorkingDir,
 		HistoryRepo:            historyRepository,
-		SnapshotsEnabled:       configs.FlagsConfig.SnapshotsEnabled,
+		FlagsConfig:            *configs.FlagsConfig,


iulianpascalau · 2023-05-04T13:33:54Z

process/sync/trieIterators/tokensSuppliesComputer.go

+			return err
+		}
+
+		tokenName := string(tokenKey)[lenESDTPrefix:]


iulianpascalau · 2023-05-04T13:35:02Z

process/sync/trieIterators/tokensSuppliesComputer_test.go

+		userAcc.SetDataTrie(&trie.TrieStub{
+			GetAllLeavesOnChannelCalled: func(leavesChannels *common.TrieIteratorChannels, ctx context.Context, rootHash []byte, keyBuilder common.KeyBuilder) error {
+				esToken := &esdt.ESDigitalToken{
+					Value: big.NewInt(37),


…ies' into MX-14120-repopulate-tokens-supplies

sstanculeanu · 2023-05-05T08:42:07Z

process/sync/trieIterators/tokensSuppliesComputer.go

+	"github.com/multiversx/mx-chain-go/trie/keyBuilder"
+)
+
+type tokensSuppliesProcessor struct {


rename file?

sstanculeanu · 2023-05-05T09:11:34Z

process/sync/trieIterators/tokensSuppliesComputer.go

+	tokenIDStr := string(tokenID)
+	if nonce > 0 {
+		t.putInSuppliesMap(string(tokenID), value) // put for collection as well
+		nonceStr := fmt.Sprintf("%d", nonce)


as discussed

Suggested change

nonceStr := fmt.Sprintf("%d", nonce)

nonceStr := hex.EncodeToString(big.NewInt(int64(nonce)).Bytes())

sstanculeanu · 2023-05-05T09:22:17Z

process/sync/trieIterators/tokensSuppliesComputer_test.go

@@ -0,0 +1,245 @@
+package trieIterators


code coverage can be increased to close to 100% on this package

sstanculeanu · 2023-05-05T09:26:07Z

process/sync/shardblock.go

 	go boot.syncBlocks(ctx)
+	return nil


add new line before?

sstanculeanu · 2023-05-05T09:30:54Z

dblookupext/esdtSupply/proto/supplyESDT.proto

@@ -12,4 +12,5 @@ message SupplyESDT {
  bytes  Supply = 1  [(gogoproto.jsontag) = "value", (gogoproto.casttypewith) = "math/big.Int;github.com/multiversx/mx-chain-core-go/data.BigIntCaster"];
  bytes  Burned = 2  [(gogoproto.jsontag) = "burned", (gogoproto.casttypewith) = "math/big.Int;github.com/multiversx/mx-chain-core-go/data.BigIntCaster"];
  bytes  Minted = 3  [(gogoproto.jsontag) = "minted", (gogoproto.casttypewith) = "math/big.Int;github.com/multiversx/mx-chain-core-go/data.BigIntCaster"];
+  bool   RecomputedSupply = 4 [(gogoproto.jsontag) = "RecomputedSupply"];


Suggested change

bool RecomputedSupply = 4 [(gogoproto.jsontag) = "RecomputedSupply"];

bool RecomputedSupply = 4 [(gogoproto.jsontag) = "recomputedSupply"];

to align with the others

good find. fixed

gabi-vuls

System test passed

bogdan-rosianu and others added 4 commits April 27, 2023 15:27

MX-14120: repopulate tokens supplies

245e527

Merge branch 'rc/v1.6.0' into MX-14120-repopulate-tokens-supplies

04c3205

MX-14120: fix linter issue

2d07755

Merge remote-tracking branch 'origin/MX-14120-repopulate-tokens-suppl…

84aef9f

…ies' into MX-14120-repopulate-tokens-supplies

bogdan-rosianu self-assigned this Apr 27, 2023

bogdan-rosianu marked this pull request as ready for review April 27, 2023 14:11

bogdan-rosianu marked this pull request as draft April 28, 2023 12:16

bogdan-rosianu and others added 4 commits April 28, 2023 17:28

MX-14120: fixes

26cd750

Merge branch 'rc/v1.6.0' into MX-14120-repopulate-tokens-supplies

231e7f9

MX-14120: adapt tests

a66b340

Merge remote-tracking branch 'origin/MX-14120-repopulate-tokens-suppl…

2f769c1

…ies' into MX-14120-repopulate-tokens-supplies

bogdan-rosianu marked this pull request as ready for review May 2, 2023 14:12

iulianpascalau self-requested a review May 2, 2023 14:27

iulianpascalau reviewed May 3, 2023

View reviewed changes

bogdan-rosianu and others added 3 commits May 3, 2023 17:10

MX-14120: fixes after review

5d57d8e

bugfix + add custom rounds per epoch in local testnet

ce1eafd

Merge branch 'rc/v1.6.0' into MX-14120-repopulate-tokens-supplies

4fd5d96

iulianpascalau reviewed May 4, 2023

View reviewed changes

bogdan-rosianu added 2 commits May 4, 2023 16:44

fix after review

13d01bd

Merge remote-tracking branch 'origin/MX-14120-repopulate-tokens-suppl…

08a2a9b

…ies' into MX-14120-repopulate-tokens-supplies

iulianpascalau previously approved these changes May 4, 2023

View reviewed changes

sstanculeanu self-requested a review May 5, 2023 08:26

sstanculeanu reviewed May 5, 2023

View reviewed changes

fixes after review

0e2ebb4

bogdan-rosianu dismissed iulianpascalau’s stale review via 0e2ebb4 May 5, 2023 12:36

Merge branch 'rc/v1.6.0' into MX-14120-repopulate-tokens-supplies

cc695a7

sstanculeanu previously approved these changes May 5, 2023

View reviewed changes

iulianpascalau previously approved these changes May 5, 2023

View reviewed changes

Merge branch 'rc/v1.6.0' into MX-14120-repopulate-tokens-supplies

0148ab0

gabi-vuls previously approved these changes May 16, 2023

View reviewed changes

fix test are merge

7029a48

bogdan-rosianu dismissed stale reviews from gabi-vuls, iulianpascalau, and sstanculeanu via 7029a48 May 16, 2023 12:36

gabi-vuls approved these changes May 16, 2023

View reviewed changes

sstanculeanu approved these changes May 16, 2023

View reviewed changes

iulianpascalau approved these changes May 16, 2023

View reviewed changes

bogdan-rosianu merged commit 360ed29 into rc/v1.6.0 May 16, 2023
5 checks passed

bogdan-rosianu deleted the MX-14120-repopulate-tokens-supplies branch May 16, 2023 13:54

schimih added the type:improvement label May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MX-14120: repopulate tokens supplies #5204

MX-14120: repopulate tokens supplies #5204

bogdan-rosianu commented Apr 27, 2023 •

edited

codecov bot commented Apr 27, 2023 •

edited

iulianpascalau May 2, 2023

iulianpascalau May 2, 2023

bogdan-rosianu May 3, 2023

iulianpascalau May 4, 2023

iulianpascalau May 2, 2023

bogdan-rosianu May 3, 2023

iulianpascalau May 3, 2023

bogdan-rosianu May 3, 2023

iulianpascalau May 4, 2023

iulianpascalau May 4, 2023

iulianpascalau May 4, 2023

bogdan-rosianu May 4, 2023

iulianpascalau May 4, 2023

iulianpascalau May 4, 2023

iulianpascalau May 4, 2023

sstanculeanu May 5, 2023

bogdan-rosianu May 5, 2023

sstanculeanu May 5, 2023

bogdan-rosianu May 5, 2023

sstanculeanu May 5, 2023

bogdan-rosianu May 5, 2023

sstanculeanu May 5, 2023

bogdan-rosianu May 5, 2023

sstanculeanu May 5, 2023

bogdan-rosianu May 5, 2023

gabi-vuls left a comment

	nonceStr := fmt.Sprintf("%d", nonce)
	nonceStr := hex.EncodeToString(big.NewInt(int64(nonce)).Bytes())

	bool RecomputedSupply = 4 [(gogoproto.jsontag) = "RecomputedSupply"];
	bool RecomputedSupply = 4 [(gogoproto.jsontag) = "recomputedSupply"];

MX-14120: repopulate tokens supplies #5204

MX-14120: repopulate tokens supplies #5204

Conversation

bogdan-rosianu commented Apr 27, 2023 • edited

Reasoning behind the pull request

Proposed changes

Testing procedure

Pre-requisites

codecov bot commented Apr 27, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabi-vuls left a comment

Choose a reason for hiding this comment

bogdan-rosianu commented Apr 27, 2023 •

edited

codecov bot commented Apr 27, 2023 •

edited