Fix partition location issue and refactoring #1977

peterlimg · 2023-01-09T11:19:50Z

Fixes

Fix partition location overflow after deleting. The last item in the last partition's location will be moved to prior parititions as a replacement, however the item's user does not know, and would try to get it with old location, it would fail either with error of item not found or location overflow (beyond the partitions num)
Fix unit64 overflow caused by addBlobber on updating allocation

Changes

Removed the outside reference of partition locations, all will be managed internally in the partition package itself.
Adjust the code to remove the using of location in blobber allocations
Refactor the partitions packge to remove redundant 'randomSelector' and unused Callback methods.
Change to bind allocation to blobbers when create new allocation, and unbind only when blobber is removed from the allocation or the allocation is finalized or canceled
Adjust related challenge ready partition
Write unit tests to cover the changes for both partititons package and storagesc

Need to be mentioned in CHANGELOG.md?

Tests

Tasks to complete before merging PR:

Ensure system tests are passing. If not Run them manually to check for any regressions 📋
Do any new system tests need added to test this change? do any existing system tests need updated? If so create a PR at 0chain/system_test
Merge your system tests PR to master AFTER merging this PR

Associated PRs (Link as appropriate):

blobber:
gosdk:
system_test:
zboxcli:
zwalletcli:
Other: ...

make it easy to use

It's not the challenge generation SC's responsibility to do this job, if it does, it should remove itself from all blobbers instead of just the selected one.

Rather than bind them on committing_connection SC, and unbind them when there's no data stored in the allocation, we connect them immediately when creating the allocation, and only unbind them when removing blobber from the allocation, or finalize/cancel the allocation. In this way, we can avoid the potential inconsistent state and avoid unnecessary add/remove operations.

## When new allocation - assert blobber allocation added - assert blobber total offer is initialized ## When update allocation - assert add blobber will add new blobber allocation - assert remove blobber will remove the blobber allocation - assert remove blobber and it's the last bound allocation, the blobber will be removed from challenge ready partition ## When receive commit connection SC - assert blobber is added to challenge ready partition ## When finalize allocation - assert the allocation is removed from all bound blobbers - assert blobber is removed from challenge ready if no allocation is bound to it

* staging: (134 commits) added docs added docs added docs Add authorizers total rewards Add validator's total rewards Add blobber total rewards Add sharders total rewards Add miners total rewards added docs fixed ambiguous field error returned struct{} skipped tests worked out comments worked out comments worked out comments Add default value to bucket_id Add bucket_id to upsert worked out comments updated dependency versions updated dependency versions upgraded lru cache ...

codecov-commenter · 2023-01-12T23:24:42Z

Codecov Report

Merging #1977 (e0ce231) into staging (f8be3bc) will increase coverage by 0.15%.
The diff coverage is 51.39%.

@@             Coverage Diff             @@
##           staging    #1977      +/-   ##
===========================================
+ Coverage    26.26%   26.42%   +0.15%     
===========================================
  Files          370      370              
  Lines        61353    61252     -101     
===========================================
+ Hits         16117    16186      +69     
+ Misses       43322    43142     -180     
- Partials      1914     1924      +10

Flag	Coverage Δ
Unit-Tests	`26.42% <51.39%> (+0.15%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ain.net/smartcontract/storagesc/benchmark_setup.go	`4.74% <0.00%> (+0.05%)`	⬆️
...e/go/0chain.net/smartcontract/storagesc/blobber.go	`43.83% <17.64%> (-0.32%)`	⬇️
...o/0chain.net/smartcontract/storagesc/allocation.go	`50.69% <20.00%> (+1.39%)`	⬆️
...de/go/0chain.net/smartcontract/storagesc/models.go	`55.75% <25.00%> (-2.79%)`	⬇️
code/go/0chain.net/smartcontract/minersc/fees.go	`42.11% <33.33%> (ø)`
...hain.net/smartcontract/partitions/partition_gen.go	`39.34% <37.03%> (+14.34%)`	⬆️
...artcontract/storagesc/challenge_ready_partition.go	`58.33% <47.36%> (+1.19%)`	⬆️
...o/0chain.net/smartcontract/partitions/partition.go	`62.65% <50.00%> (+4.22%)`	⬆️
...smartcontract/storagesc/blobber_alloc_partition.go	`47.05% <50.00%> (+4.95%)`	⬆️
...o/0chain.net/smartcontract/storagesc/models_gen.go	`34.19% <50.00%> (-0.98%)`	⬇️
... and 21 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

dabasov

lgtm

# Conflicts: # code/go/0chain.net/smartcontract/storagesc/challenge_test.go

peterlimg added 28 commits January 4, 2023 21:08

Add partition item location map in partitions itself to

c3f2579

make it easy to use

Remove outside partitions location

d1fc415

Add partitions.Foreach to iterate through all items in partititons

c0bd2fd

Enhance partitions

5c79915

Cache partition index keys fix

6a0beef

Debug active pool pending

65ca06a

Fix part.Foreach saving

23d8fe2

Remove unused partition CallBack and AddRand

a8d60b3

Add debug info incase empty last partition error occurs

ce0e930

Assert partitions.NumPartitions and len(partitions) match

af2967a

Rename notFound to notFoundIndex

b1449ff

Remove redundant randomSelector

0b22d3b

Fix and add fuzz tests to partitions Add/Remove

82f3732

Fix load location

2d8e265

More tests and location load fix

0c79f2d

Test partitions.UpdateItem

423c6e9

Add test cases for partitions.Update

ad27f13

Add tests for GetRandomItems

0d18839

Add more tests

1124a66

Remove location from Add return variables

0bbf075

Do not unbind allocation and blobber when allocation is expired

644befc

It's not the challenge generation SC's responsibility to do this job, if it does, it should remove itself from all blobbers instead of just the selected one.

Remove challenge ready process from removeAllocationFromBlobber

6745157

Adjust partitions methods usage

15390ea

Fix uint64 overflow when update allocation to add new blobber

19e9d9d

Fix unit tests

f8166f6

Update partitions README

f0e4923

peterlimg requested review from dabasov and guruhubb as code owners January 9, 2023 11:19

peterlimg added 2 commits January 9, 2023 22:28

Fix unit test

e433630

peterlimg and others added 2 commits January 15, 2023 21:10

Merge branch 'staging' into fix/partitions-loc

a665217

Merge branch 'staging' into fix/partitions-loc

95ae56a

dabasov approved these changes Jan 21, 2023

View reviewed changes

Merge branch 'staging' into fix/partitions-loc

e0ce231

# Conflicts: # code/go/0chain.net/smartcontract/storagesc/challenge_test.go

peterlimg merged commit 31c960f into staging Jan 22, 2023

peterlimg deleted the fix/partitions-loc branch January 22, 2023 03:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix partition location issue and refactoring #1977

Fix partition location issue and refactoring #1977

peterlimg commented Jan 9, 2023

codecov-commenter commented Jan 12, 2023 •

edited

Loading

dabasov left a comment

Fix partition location issue and refactoring #1977

Fix partition location issue and refactoring #1977

Conversation

peterlimg commented Jan 9, 2023

Fixes

Changes

Need to be mentioned in CHANGELOG.md?

Tests

Associated PRs (Link as appropriate):

codecov-commenter commented Jan 12, 2023 • edited Loading

Codecov Report

dabasov left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jan 12, 2023 •

edited

Loading