Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix checking storage over randomly ordered list #1156

Merged
merged 9 commits into from
Aug 18, 2021

Conversation

janezpodhostnik
Copy link
Contributor

When looping through touched addresses for the purposes of checking storage, the code returns early. The list of touche addresses is acquired from a map of addresses, and thus randomly sorted.

This caused that when we were checking storage and there was a error before the end of the loop, some random addresses were not touched.

This could potentially fail verification at the same step.

@codecov-commenter
Copy link

codecov-commenter commented Aug 17, 2021

Codecov Report

Merging #1156 (58a6550) into master (82c2c48) will increase coverage by 0.03%.
The diff coverage is 45.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1156      +/-   ##
==========================================
+ Coverage   53.37%   53.40%   +0.03%     
==========================================
  Files         324      324              
  Lines       21944    21955      +11     
==========================================
+ Hits        11712    11726      +14     
+ Misses       8639     8634       -5     
- Partials     1593     1595       +2     
Flag Coverage Δ
unittests 53.40% <45.45%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
fvm/state/state.go 79.27% <0.00%> (-4.54%) ⬇️
fvm/transactionStorageLimiter.go 78.94% <71.42%> (-6.77%) ⬇️
cmd/util/ledger/migrations/storage_v4.go 42.16% <0.00%> (+0.60%) ⬆️
...sus/approvals/assignment_collector_statemachine.go 51.92% <0.00%> (+9.61%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82c2c48...58a6550. Read the comment docs.

Copy link
Contributor

@huitseeker huitseeker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ack, though I'm not the most familiar with that area.

fvm/state/state.go Outdated Show resolved Hide resolved
for k := range s.updatedAddresses {
addresses = append(addresses, k)
i := sort.Search(len(addresses), func(i int) bool { return addresses[i].Hex() >= k.Hex() })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid this will be used a lot (for every transaction?) and using .Hex() isn't best performance-wide. bytes.Compare instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to suggest similar thing, using bytes.Compare over slices or conversion to uint64 and compare considering the number of bytes of addresses

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather do byte comparison... converting to uint introduces an element that could fail.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

converting to uint introduces an element that could fail.

Not really - one of the Go quirks I learned recently, it will silently truncate the value https://golang.org/ref/spec#Conversions

fvm/state/state.go Outdated Show resolved Hide resolved
}

sort.Sort(addresses)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any performance concern with the Sort?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list is usually small and the comparison is with byte slices so the performance cost should be small. however, the list needs to be sorted...

This also only happens once per transaction.

Copy link
Contributor Author

@janezpodhostnik janezpodhostnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having a lot of issues getting this through CI

@huitseeker
Copy link
Contributor

bors merge

bors bot added a commit that referenced this pull request Aug 18, 2021
1156: Fix checking storage over randomly ordered list r=huitseeker a=janezpodhostnik

When looping through touched addresses for the purposes of checking storage, the code returns early. The list of touche addresses is acquired from a map of addresses, and thus randomly sorted.

This caused that when we were checking storage and there was a error before the end of the loop, some random addresses were not touched.

This could potentially fail verification at the same step.

Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>
Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com>
@bors
Copy link
Contributor

bors bot commented Aug 18, 2021

Build failed:

@janezpodhostnik
Copy link
Contributor Author

bors merge

bors bot added a commit that referenced this pull request Aug 18, 2021
1156: Fix checking storage over randomly ordered list r=janezpodhostnik a=janezpodhostnik

When looping through touched addresses for the purposes of checking storage, the code returns early. The list of touche addresses is acquired from a map of addresses, and thus randomly sorted.

This caused that when we were checking storage and there was a error before the end of the loop, some random addresses were not touched.

This could potentially fail verification at the same step.

Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>
Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com>
@bors
Copy link
Contributor

bors bot commented Aug 18, 2021

Build failed:

@janezpodhostnik
Copy link
Contributor Author

bors merge

bors bot added a commit that referenced this pull request Aug 18, 2021
1156: Fix checking storage over randomly ordered list r=janezpodhostnik a=janezpodhostnik

When looping through touched addresses for the purposes of checking storage, the code returns early. The list of touche addresses is acquired from a map of addresses, and thus randomly sorted.

This caused that when we were checking storage and there was a error before the end of the loop, some random addresses were not touched.

This could potentially fail verification at the same step.

Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>
Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com>
@bors
Copy link
Contributor

bors bot commented Aug 18, 2021

Build failed:

@janezpodhostnik janezpodhostnik merged commit 278b187 into master Aug 18, 2021
@janezpodhostnik janezpodhostnik deleted the janez/fix-iterating-over-unordered-list branch August 18, 2021 19:27
janezpodhostnik added a commit that referenced this pull request Aug 20, 2021
janezpodhostnik added a commit that referenced this pull request Aug 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants