Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attestation aggregation: remove proper set duplicates #8063

Merged
merged 10 commits into from
Dec 11, 2020

Conversation

farazdagi
Copy link
Contributor

@farazdagi farazdagi commented Dec 7, 2020

What type of PR is this?

Bug fix

What does this PR do? Why is it needed?

  • Deduplicate aggregated attestations that contain some attestations which are nothing more but proper sets of other attestations (thus do not have any new info).
  • Such proper set attestations can be removed, but only in situations where we do not plan to do any more aggregation, since if we do -- some profitable (ones that produce profitable aggregation) intermediary attestations, even if they are proper sets, might be removed. So, this deduplication must take place as late in the process as possible, hence it is implemented in proposer.

Which issues(s) does this PR fix?

Part of #7871

Other notes for review

  • Extracted from Attestation aggregation: optimizations and benchmarks #7938
  • In attestations/kv/aggregated.go we make sure that we do not add redundant bits (see attestations/kv/seen_bits.go). However, while our method is good enough to make sure that non-overlapping attestations all have some unseen bits present, after the aggregation situation can change (so, we still need to make sure we deduplicate before proposing).

Here is a sample case:

  • Suppose we have the following two atts in database:
1001.0001
1100.0000

So, seen bits are: 1101.0001.

Now, another aggregate comes in: 0100.0010. It will not be marked as seen as it introduces new info at 0000.00X0 position. So, pool will be updated to:

1001.0001
1100.0000
0100.0010

On aggregation, it will become:

1101.0011 (1001.0001 + 0100.0010)
1100.0000

And, clearly, 1100.0000 is redundant as it is a proper subset of the first aggregate, thus needs to be deduplicated.

@farazdagi farazdagi added this to the v1.0.5 milestone Dec 7, 2020
@farazdagi farazdagi self-assigned this Dec 7, 2020
@farazdagi farazdagi marked this pull request as ready for review December 10, 2020 20:36
@farazdagi farazdagi requested a review from a team as a code owner December 10, 2020 20:36
@farazdagi farazdagi added the Ready For Review A pull request ready for code review label Dec 10, 2020
@@ -2105,27 +2105,6 @@ func TestProposer_DeleteAttsInPool_Aggregated(t *testing.T) {
assert.Equal(t, 0, len(atts), "Did not delete unaggregated attestation")
}

func TestProposer_SortProfitableAtts(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason this test is removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not removed, it is moved (proposer_utils_test.go file was added, and this test, correctly, was moved there).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atts[j] = atts[len(atts)-1]
atts[len(atts)-1] = nil
atts = atts[:len(atts)-1]
j--
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you missing a break after this line?

Copy link
Contributor Author

@farazdagi farazdagi Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, definitely not.

We fixate on atts[i] and can remove as many atts[j] w/o breaking (I am swapping the current element for the last and decrease the len -- common idiom). I decrement j so that swapped element is checked too.

The break in the second else if is because when we need to remove atts[i] (and we fixate on it), we swap the last and the current element, and need to break, and check that last element's index once again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense!

@farazdagi farazdagi merged commit 46d99fd into develop Dec 11, 2020
@delete-merged-branch delete-merged-branch bot deleted the maxcover-fix-properset-issue branch December 11, 2020 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ready For Review A pull request ready for code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants