Skip to content

Conversation

@michaelsproul
Copy link
Member

Issue Addressed

Closes #1719

Proposed Changes

Lift the internal RwLocks and Mutexes from the Observed* data structures to resolve the race conditions described in #1719.

Most of this work was done by @paulhauner on his lift-locks branch, I merely updated it for the current master and checked over it.

Additional Info

I think it would be prudent to test this on a testnet or two before mainnet launch, just to be sure that the extra lock contention doesn't negatively impact performance.

Squashed commit of the following:

commit dc76a33
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 11:37:48 2020 +1000

    Tighten `BeaconChain` field visibility

commit 7c60abd
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 10:45:26 2020 +1000

    Fix test compile errors

commit f726abb
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 10:26:32 2020 +1000

    Lift locks for observed operations

commit 2332366
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 10:14:54 2020 +1000

    Lift locks for observed producers

commit a99a2dc
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 07:57:47 2020 +1000

    Lift observed attesters locks

commit 6093eb9
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 07:51:08 2020 +1000

    Add more debug logs for block proc.

commit 3331751
Author: Paul Hauner <paul@paulhauner.com>
Date:   Thu Aug 6 07:37:56 2020 +1000

    Lift ObservedAttestations locks

commit 0f93f9c
Author: Paul Hauner <paul@paulhauner.com>
Date:   Wed Aug 5 15:15:42 2020 +1000

    Push naive attns into op pool when producing block

commit 4b128ee
Author: Paul Hauner <paul@paulhauner.com>
Date:   Wed Aug 5 15:06:22 2020 +1000

    Lift RwLocks out of naive pool
@michaelsproul michaelsproul added work-in-progress PR is a work-in-progress and removed ready-for-review The code is ready for review labels Nov 20, 2020
@michaelsproul michaelsproul added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Nov 20, 2020
Copy link
Member

@paulhauner paulhauner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm, so much easier to reason about! Thank you!

I think it would be prudent to test this on a testnet or two before mainnet launch, just to be sure that the extra lock contention doesn't negatively impact performance.

Agree, Pyrmont should be good for that I think. I've run it locally and can't notice a change in CPU usage.

@michaelsproul
Copy link
Member Author

bors r+

bors bot pushed a commit that referenced this pull request Nov 22, 2020
## Issue Addressed

Closes #1719

## Proposed Changes

Lift the internal `RwLock`s and `Mutex`es from the `Observed*` data structures to resolve the race conditions described in #1719.

Most of this work was done by @paulhauner on his `lift-locks` branch, I merely updated it for the current `master` and checked over it.

## Additional Info

I think it would be prudent to test this on a testnet or two before mainnet launch, just to be sure that the extra lock contention doesn't negatively impact performance.
@bors
Copy link

bors bot commented Nov 22, 2020

@bors bors bot changed the title Fix race condition in seen caches [Merged by Bors] - Fix race condition in seen caches Nov 22, 2020
@bors bors bot closed this Nov 22, 2020
@michaelsproul michaelsproul deleted the racy-seen-caches branch November 23, 2020 00:08
@michaelsproul michaelsproul added the consensus An issue/PR that touches consensus code, such as state_processing or block verification. label Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A0 consensus An issue/PR that touches consensus code, such as state_processing or block verification. ready-for-review The code is ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Racy seen caches

3 participants