Convert the type of Quota and QuotaUsed from big.Int to uint64 #654

DhunterAO · 2020-07-04T09:54:20Z

The uint64 type introduce the risk of overflow, so I add checks after add and sub operations, please check whether they are correct or if there are some checks missed.
The update of storage may cause hard fork of the blockchain, so I only update the type in the memory layer and keep the big.Int in storage layer. Please check whether there is any code in storage influnced.

Convert the type of Quota and QuotaUsed from big.Int to uint64, which can accelerate the system.

…us into uint64Quota

ucwong · 2020-07-04T10:04:36Z

consensus/cuckoo/consensus.go

@@ -301,7 +301,7 @@ func (cuckoo *Cuckoo) verifyHeader(chain consensus.ChainReader, header, parent *
 		return consensus.ErrInvalidNumber
 	}

-	if header.Quota.Cmp(new(big.Int).Add(parent.Quota, new(big.Int).SetUint64(chain.Config().GetBlockQuota(header.Number)))) != 0 {
+	if header.Quota < parent.Quota || header.Quota != parent.Quota+chain.Config().GetBlockQuota(header.Number) {


Whether the second condition contains first one?

The first one consider the overflow of uint64.
For example, parent.Quota = MaxUint64 - 1, blockQuota = 2, header.Quota = 1. This situation satisfies the second condition but dissatisfies the first one.

Can quota reach MaxUint64, we should avoid overflow in quota calculation.
In fact we can calculate when the quota limit will be

Yeah, gasLimit is set for each block own, but the quota is accumulated through the whole blockchain.
For the max quota value in each block is 65536 * 128 (for dolores), which means the limit will be reached at block number 2199023255551

Yes, gas limit from big.int to uint64 because they know the ceiling ethereum/go-ethereum@6f69cdd#diff-afbbe5ab9727e568254cfee647e62876

2199023255551 is huge for block number, should we consider it as ∞ ?

If we can consider it as ∞, I think we can also convert the quota into uint64

ucwong · 2020-07-04T10:08:22Z

consensus/cuckoo/consensus.go

-	header.QuotaUsed = new(big.Int).Set(parent.QuotaUsed)
+	header.Quota = parent.Quota + chain.Config().GetBlockQuota(header.Number)
+	if header.Quota < parent.Quota {
+		panic("quota reaches the upper limit of uint64")


Is there any possible to be panic here, if there is, big.Int should be more suitable

Due to the consideration of overflow, if the result of addition exceeds the max of uint64, the panic will occur.
If the type is big.Int, it seems to be unlimited.

Yes, the big.Int is used for avoid overflow. If it can be overflow , big.Int should be suitable for quota

If a valiate block satisfy this condition occurs, all the nodes will crash.

we should confirm all the panics in code can't be attacked.

In fact, I just test that the addition overflow will panic automatically in Golang. Not like that in C/C++. So we need to do the check before to prevent the panics.

codecov · 2020-07-04T10:09:02Z

Codecov Report

Merging #654 into master will decrease coverage by 0.01%.
The diff coverage is 26.31%.

@@            Coverage Diff             @@
##           master     #654      +/-   ##
==========================================
- Coverage   50.85%   50.84%   -0.02%     
==========================================
  Files         405      407       +2     
  Lines       51727    51755      +28     
==========================================
+ Hits        26308    26316       +8     
- Misses      23405    23415      +10     
- Partials     2014     2024      +10

Impacted Files	Coverage Δ
common/math/min.go	`0.00% <0.00%> (ø)`
consensus/cuckoo/consensus.go	`1.71% <0.00%> (-0.01%)`	⬇️
core/blockchain_insert.go	`79.31% <ø> (ø)`
core/state_prefetcher.go	`34.37% <0.00%> (-2.30%)`	⬇️
core/types/block.go	`25.60% <0.00%> (+0.60%)`	⬆️
core/types/gen_header_json.go	`0.00% <0.00%> (ø)`
ctxc/api_tracer.go	`0.00% <0.00%> (ø)`
core/state_transition.go	`49.35% <26.66%> (ø)`
core/state_processor.go	`67.85% <33.33%> (-2.52%)`	⬇️
core/quotapool.go	`38.88% <38.88%> (ø)`
... and 13 more

ucwong · 2020-07-04T10:18:08Z

core/state_transition.go


-		st.state.SubUpload(st.to(), quota) //64 ~ 1024 bytes
+		st.state.SubUpload(st.to(), big.NewInt(int64(quota))) //64 ~ 1024 bytes


big.NewInt(0).SetUint64(value)
The upload also can be changed to uint64 for it should always below the quota, but it maybe complex, we will think about it in the future

ucwong · 2020-07-04T10:21:16Z

Did you use go generate to generate code like gen_*

DhunterAO · 2020-07-04T10:23:46Z

Did you use go generate to generate code like gen_*

I havn't installed the gencodec, so just modify the gen_header_json.go manually.

ucwong · 2020-07-04T10:31:58Z

It's great, what we should think about is the overflow of uint64 in this situation

ucwong · 2020-07-04T10:55:58Z

I have built it and let it keep running for some days and see the metrics chart

DhunterAO · 2020-07-04T11:00:50Z

got it

ucwong · 2020-07-04T11:14:41Z

ethereum/go-ethereum#15466

ucwong · 2020-07-06T03:13:51Z

Before and after this PR to fully sync blockchain (alloc/s)

ucwong

LGTM

DhunterAO · 2020-07-07T09:08:29Z

Before and after this PR to fully sync blockchain (alloc/s)

Looks cool. I wonder what each color means? Does this PR increase the alloc number per second?

ucwong · 2020-07-07T09:12:51Z

Before and after this PR to fully sync blockchain (alloc/s)

Looks cool. I wonder what each color means? Does this PR increase the alloc number per second?

Different scopes 1m 5m 15m and mean

DhunterAO and others added 5 commits July 4, 2020 17:43

update the type of Quota and QuotaUsed

e49ecb1

Convert the type of Quota and QuotaUsed from big.Int to uint64, which can accelerate the system.

Update quotapool.go

ae1c7ca

update quota in block test

500cb09

fix quota and quotaUsed type overrides

395780f

Merge branch 'uint64Quota' of github.com:CortexFoundation/CortexThese…

32ce85e

…us into uint64Quota

ucwong reviewed Jul 4, 2020

View reviewed changes

ucwong requested a review from ShadowErii July 4, 2020 10:47

ucwong mentioned this pull request Jul 4, 2020

Current code review #653

Open

ucwong requested review from wlt-cortex, declmal, lizhencortex and zkh2018 July 4, 2020 10:52

ucwong added 4 commits July 4, 2020 21:40

Prepare quota pool before apply tx

6bd337e

Negative/overflow quota fix

d6671b6

quota uint64 to big.Int

3db56d3

Prepare quota pool before precacheTransaction

4b8b015

ucwong self-requested a review July 7, 2020 06:41

ucwong approved these changes Jul 7, 2020

View reviewed changes

ucwong merged commit 259994e into master Jul 7, 2020

DhunterAO deleted the uint64Quota branch July 7, 2020 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert the type of Quota and QuotaUsed from big.Int to uint64 #654

Convert the type of Quota and QuotaUsed from big.Int to uint64 #654

DhunterAO commented Jul 4, 2020

ucwong Jul 4, 2020

DhunterAO Jul 4, 2020

ucwong Jul 4, 2020

DhunterAO Jul 4, 2020

ucwong Jul 4, 2020 •

edited

DhunterAO Jul 4, 2020

ucwong Jul 4, 2020 •

edited

DhunterAO Jul 4, 2020

ucwong Jul 4, 2020

ucwong Jul 4, 2020 •

edited

ucwong Jul 4, 2020

DhunterAO Jul 4, 2020

codecov bot commented Jul 4, 2020 •

edited

ucwong Jul 4, 2020 •

edited

DhunterAO Jul 4, 2020

DhunterAO Jul 4, 2020

ucwong commented Jul 4, 2020

DhunterAO commented Jul 4, 2020

ucwong commented Jul 4, 2020 •

edited

ucwong commented Jul 4, 2020

DhunterAO commented Jul 4, 2020

ucwong commented Jul 4, 2020

ucwong commented Jul 6, 2020 •

edited

ucwong left a comment

DhunterAO commented Jul 7, 2020

ucwong commented Jul 7, 2020


		st.state.SubUpload(st.to(), quota) //64 ~ 1024 bytes
		st.state.SubUpload(st.to(), big.NewInt(int64(quota))) //64 ~ 1024 bytes

Convert the type of Quota and QuotaUsed from big.Int to uint64 #654

Convert the type of Quota and QuotaUsed from big.Int to uint64 #654

Conversation

DhunterAO commented Jul 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ucwong Jul 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ucwong Jul 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ucwong Jul 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 4, 2020 • edited

Codecov Report

ucwong Jul 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ucwong commented Jul 4, 2020

DhunterAO commented Jul 4, 2020

ucwong commented Jul 4, 2020 • edited

ucwong commented Jul 4, 2020

DhunterAO commented Jul 4, 2020

ucwong commented Jul 4, 2020

ucwong commented Jul 6, 2020 • edited

ucwong left a comment

Choose a reason for hiding this comment

DhunterAO commented Jul 7, 2020

ucwong commented Jul 7, 2020

ucwong Jul 4, 2020 •

edited

ucwong Jul 4, 2020 •

edited

ucwong Jul 4, 2020 •

edited

codecov bot commented Jul 4, 2020 •

edited

ucwong Jul 4, 2020 •

edited

ucwong commented Jul 4, 2020 •

edited

ucwong commented Jul 6, 2020 •

edited