Implement ListBeaconCommittees RPC Method #3977

rauljordan · 2019-11-11T23:12:16Z

Description

Write why you are making the changes in this pull request

This PR implements the ListBeaconCommittees method in our RPC package, allowing for easy retrieval of all committees in an epoch for the usage of slashing detection. Thanks @shayzluf for suggesting this feature. It fetches all beacon committees, paginated by default, based on input parameters of epoch as a query filter.

Using this functionality and an attestation's bitfield, we can convert attestations into indexed form, allowing for easier slashing detection!

Link anything that would be helpful or relevant to the reviewers

Needs tests
Needs to be tested at runtime with archival on/off

The request/response type protobuf messages are given below:

message ListCommitteesRequest {
    oneof query_filter {
        // Optional criteria to retrieve data at a specific epoch.
        uint64 epoch = 1;

        // Optional criteria to retrieve genesis data.
        bool genesis = 2;
    }

    // The maximum number of Validators to return in the response.
    // This field is optional.
    int32 page_size = 3;

    // A pagination token returned from a previous call to `GetValidators`
    // that indicates where this listing should continue from.
    // This field is optional.
    string page_token = 4;
}

message BeaconCommittees {
    message CommitteeItem {
        // A committee of validator indices that need to attest to beacon blocks.
        repeated uint64 committee = 1;

        // The slot at which the committee is assigned to.
        uint64 slot = 2;
    }
    // The epoch for which the committees in the response belong to.
    uint64 epoch = 1;

    // A list of committees of validators for given  epoch.
    repeated CommitteeItem committees = 2;

    // The number of active validators at the given epoch.
    uint64 active_validator_count = 3;

    // A pagination token returned from a previous call
    // that indicates from where the listing should continue.
    string next_page_token = 4;

    // Total count of committees matching the request filter.
    int32 total_size = 5;
}

* rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes

* Starting * Add AggregateAndProof

* rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Unskip block util tests

* Starting * Add AggregateAndProof * Unskip slot processing mainnet test

* Rm outdated interop tests * Rm test runner * Gazelle

…v0.9

* Updated committee cache * Removed shuffled indices cache * Started testing run time * Lint * Fixed test

rauljordan · 2019-11-12T20:57:14Z

beacon-chain/archiver/service.go

@@ -69,10 +69,6 @@ func (s *Service) Status() error {
 // We archive committee information pertaining to the head state's epoch.
 func (s *Service) archiveCommitteeInfo(ctx context.Context, headState *pb.BeaconState) error {
 	currentEpoch := helpers.SlotToEpoch(headState.Slot)
-	committeeCount, err := helpers.CommitteeCountAtSlot(headState, helpers.StartSlot(currentEpoch))


This is wrong, we shouldn't store this because it can change on a slot by slot basis

rauljordan · 2019-11-12T20:57:31Z

beacon-chain/archiver/service.go

-		ProposerSeed:   proposerSeed[:],
-		AttesterSeed:   attesterSeed[:],
-		CommitteeCount: committeeCount * params.BeaconConfig().SlotsPerEpoch,
+		ProposerSeed: proposerSeed[:],


These two seeds are really all we need to determine historical assignments alongside historical balances

beacon-chain/archiver/service.go

rauljordan · 2019-11-12T20:58:07Z

beacon-chain/rpc/beacon/committees.go

+	var attesterSeed [32]byte
+	var activeIndices []uint64
+	var err error
+	if requestingGenesis || helpers.SlotToEpoch(startSlot) < helpers.SlotToEpoch(headState.Slot) {


This is the archival condition

rauljordan · 2019-11-12T20:58:17Z

beacon-chain/rpc/beacon/committees.go

+			)
+		}
+		attesterSeed = bytesutil.ToBytes32(archivedCommitteeInfo.AttesterSeed)
+	} else if !requestingGenesis && helpers.SlotToEpoch(startSlot) == helpers.SlotToEpoch(headState.Slot) {


Otherwise if requesting current epoch, return current results

Can you add these as comments? it seems helpful

beacon-chain/rpc/beacon/committees.go

…m into beacon-committees-rpc

prestonvanloon · 2019-11-12T23:08:58Z

beacon-chain/rpc/service_test.go

@@ -42,30 +42,6 @@ func TestLifecycle_OK(t *testing.T) {

 }

-func TestRPC_BadEndpoint(t *testing.T) {


This looks unrelated, but it was so flaky and annoying that I support removing it in this PR.

It also wasn't really testing much - it wasn't really behavior we cared about testing in the first place

prestonvanloon · 2019-11-12T23:17:56Z

beacon-chain/rpc/beacon/committees.go

+			)
+		}
+		attesterSeed = bytesutil.ToBytes32(archivedCommitteeInfo.AttesterSeed)
+	} else if !requestingGenesis && helpers.SlotToEpoch(startSlot) == helpers.SlotToEpoch(headState.Slot) {


Can you add these as comments? it seems helpful

nisdas · 2019-11-13T05:26:20Z

beacon-chain/rpc/beacon/committees.go

+//
+// If no filter criteria is specified, the response returns
+// all beacon committees for the current epoch. The results are paginated by default.
+func (bs *Server) ListBeaconCommittees(


For this we might need more options for the req *ethpb.ListCommitteesRequest. Along with epoch we also need to give a query option, where we can provide the state root.

In the event of a fork in the network, you could have two chains with different committees at the same epoch. To differentiate on the two sets of committees we would need to be able to identify committees from forked chains, so the state root is a good way to identify them. @shayzluf and I discussed this offline, in the event of a fork in the network , we need to be able to get the appropriate committees from each forked chain.

Great point, that's a neat suggestion. So what happens if you provide a state root that doesnt match head state?

Maybe we can do this in a follow-up PR as that will increase the diff quite a lot.

I guess it would mean we are asking for the committees from a forked chain :) This would be important for the slasher, as it needs to be able to detect malicious attestations in the event of a fork. Sounds good, we can do it in a follow up PR

terencechain and others added 30 commits October 29, 2019 08:44

Update seed domains (#3872)

b3714e8

Remove Transfers (#3870)

8eebd52

Remove active index roots and compact committee roots (#3869)

b6645d6

Update inclusion reward (#3886)

cf169c8

Alter proposer selection logic (#3884)

6294037

Fix early committee bias (#3888)

d2f4b2b

Remove shards and committees (#3896)

c864627

Epoch spec tests v0.9 (#3907)

2c88364

Update deposit contract (#3906)

59f528f

Proto spec tests v0.9 (#3908)

9bc9b91

* Starting * Add AggregateAndProof

Slot processing spec test V0.9 (#3912)

064e963

* Starting * Add AggregateAndProof * Unskip slot processing mainnet test

Unskip minimal spec test for finalization (#3920)

5d2f142

Remove outdated interop tests (#3922)

842a5ac

* Rm outdated interop tests * Rm test runner * Gazelle

Update validator to use proposer slot (#3919)

4e22ca2

Fix committee assignment (#3931)

8617344

Replace shard with committee index (#3930)

b20fc61

Conflict

03f2cb0

Conflict

6806e33

Clean up (#3933)

a35da60

Remove shard filter in db (#3936)

d5c2e0c

Merge branch 'master' into v0.9

4dab4c0

Merge branch 'master' into v0.9

53c4c18

Remove lightouse compatibility test (#3939)

45a88bb

Merge branch 'master' of https://github.com/prysmaticlabs/prysm into …

a2814d6

…v0.9

Merge branch 'master' of github.com:prysmaticlabs/prysm into v0.9

d27bd4e

Merge branch 'master' of https://github.com/prysmaticlabs/prysm into …

29b6d18

…v0.9

Merge branch 'v0.9' of https://github.com/prysmaticlabs/prysm into v0.9

4b4a761

Update Committee Cache for v0.9 (#3948)

bb64762

* Updated committee cache * Removed shuffled indices cache * Started testing run time * Lint * Fixed test

rauljordan added 2 commits November 12, 2019 14:34

test resolution

2fbc45a

ux improvements and bug fixes

4100411

rauljordan commented Nov 12, 2019

View reviewed changes

rauljordan added 4 commits November 12, 2019 15:01

complete

370db85

comments

0bf6266

Update beacon-chain/archiver/service.go

fc979b6

Update beacon-chain/rpc/beacon/committees.go

37f9b74

rauljordan requested review from terencechain, nisdas, prestonvanloon, 0xKiwi and shayzluf November 12, 2019 21:05

0xKiwi previously approved these changes Nov 12, 2019

View reviewed changes

rauljordan added 2 commits November 12, 2019 15:13

elim bad test

3ead960

Merge branch 'beacon-committees-rpc' of github.com:prysmaticlabs/prys…

7ea385a

…m into beacon-committees-rpc

rauljordan dismissed 0xKiwi’s stale review via 7ea385a November 12, 2019 21:13

prestonvanloon reviewed Nov 12, 2019

View reviewed changes

rauljordan added 2 commits November 12, 2019 23:13

preston feedback

fc526b0

Merge branch 'master' into beacon-committees-rpc

f09c38b

nisdas reviewed Nov 13, 2019

View reviewed changes

nisdas approved these changes Nov 13, 2019

View reviewed changes

rauljordan mentioned this pull request Nov 13, 2019

Provide StateRoot as Parameter to ListBeaconCommittees RPC #3991

Closed

rauljordan merged commit 75521ff into master Nov 13, 2019

delete-merged-branch bot deleted the beacon-committees-rpc branch November 13, 2019 05:32

rauljordan mentioned this pull request Nov 13, 2019

Verify archival mode works with v0.9 #3938

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement ListBeaconCommittees RPC Method #3977

Implement ListBeaconCommittees RPC Method #3977

rauljordan commented Nov 11, 2019 •

edited

Loading

rauljordan Nov 12, 2019

rauljordan Nov 12, 2019

rauljordan Nov 12, 2019

rauljordan Nov 12, 2019

prestonvanloon Nov 12, 2019

prestonvanloon Nov 12, 2019

rauljordan Nov 13, 2019

prestonvanloon Nov 12, 2019

nisdas Nov 13, 2019

rauljordan Nov 13, 2019

rauljordan Nov 13, 2019

nisdas Nov 13, 2019

		@@ -42,30 +42,6 @@ func TestLifecycle_OK(t *testing.T) {

		}

		func TestRPC_BadEndpoint(t *testing.T) {

Implement ListBeaconCommittees RPC Method #3977

Implement ListBeaconCommittees RPC Method #3977

Conversation

rauljordan commented Nov 11, 2019 • edited Loading

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rauljordan commented Nov 11, 2019 •

edited

Loading