Make API friendly to block explorers #702

paulhauner · 2019-12-11T01:11:26Z

Issue Addressed

Resolves #694
Resolves #713

Proposed Changes

Point-by-point addressing of @ppratscher's requests:

The blocks object should also include the block root hash alongside the state root hash and parent root hash

This has been done, see /beacon/block.

Currently I could not find any call that returns the current balance of a validator, it would be great if it is included in the get_beacon_validators response alongside the effective balance

The /beacon/validators endpoint now includes the balance.

The get_beacon_validators should include the validator index alongside the validator public key

The /beacon/validators endpoint now includes the validator index.

Add a get_beacon_validators_duties that returns the duties of all validators for a given epoch, otherwise we will have to call get_beacon_validator_duties with all public keys which might be not that performant (but I haven't tested this up to now)

The /beacon/validators/all endpoint serves this request. It's returning quite quickly on my end (40ms).

In order to de-aggregate attestations and associate an attestation with the included validators we will need the beacon committees for each epoch. I couldn't find any call that returns the committees by epoch

The /beacon/committees endpoint should fulfill this request.

In order to show participation statistics for each epoch add a call that shows the eligible ether, voted ether and the participation rate for each epoch

The consensus/global_votes endpoint should serve this request.

eligible ether = previous_epoch_active_gwei
voted ether = previous_epoch_attesting_gwei
participation rate = voted ether / eligible ether (I didn't see the need to include this in the endpoint)

Notes

~~This is a work-in-progress.~~

paulhauner · 2019-12-12T09:05:03Z

@ppratscher, I have finished implementing the endpoints you were after and I have addressed each point in the top of this PR.

I will move onto unit testing next, however I have tested the endpoints by hand. If you are eager, feel free to start building off this branch. Otherwise, feel free to wait until it has been merged to master.

peterbitfly · 2019-12-12T09:08:02Z

@paulhauner thanks a lot for your efforts! we will be starting next week on Monday to add a compatibility layer to beaconcha.in in order to support lighthouse. Will report back in case we run into any issues.

pscott · 2019-12-15T12:47:21Z

Just read the docs in this PR, everything looked good to me! Disappointed I couldn't find a single typo... :p

michaelsproul

Looks good overall, would just like to discuss some of the points raised about the /consensus end point.

michaelsproul · 2019-12-16T01:39:30Z

book/src/http_beacon.md

+
+### Returns
+
+Returns an object containing a single [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/v0.9.2/specs/core/0_beacon-chain.md#beaconblock) and it's signed root.


book/src/http_consensus.md

michaelsproul · 2019-12-16T01:39:30Z

book/src/http_consensus.md

+- `previous_epoch_active_gwei`: as above, but during the previous epoch.
+- `current_epoch_attesting_gwei`: the total staked gwei that had one or more
+    attestations included in a block during the current epoch (multiple
+	attestations by the same validator does not increase this figure).


does not -> do not?

michaelsproul · 2019-12-16T01:39:30Z

book/src/http_consensus.md

+	figure must be equal to or less than `current_epoch_attesting_gwei`.
+- `previous_epoch_attesting_gwei`: see `current_epoch_attesting_gwei`.
+- `previous_epoch_target_attesting_gwei`: see `current_epoch_target_attesting_gwei`.
+- `previous_epoch_head_attesting_gwei`: see `previous_epoch_head_attesting_gwei`.


I guess this should read "see current_epoch_head_attesting_gwei", and current_epoch_attesting_gwei should have a description above. Would be nice to arrange all the current_ fields into one block of bullet points, and follow them with all the "see above" previous_ fields.

book/src/http_consensus.md

michaelsproul · 2019-12-16T01:39:30Z

book/src/http_network.md

+
+## `/network/peers`
+
+Requests the beacon node for one `MultiAddr` for each connected peer.


Gramatically awkward, maybe:

Requests one MultiAddr for each peer connected to the beacon node

michaelsproul · 2019-12-16T01:39:30Z

book/src/http_network.md

+
+## `network/enr`
+
+Requests the beacon node for it's listening `ENR` address.


pawanjay176 · 2019-12-16T05:09:54Z

book/src/http_consensus.md

+Method | GET
+JSON Encoding | Object
+Query Parameters | `epoch`


This should be a POST method with query params None.

pawanjay176 · 2019-12-16T05:15:07Z

book/src/http_consensus.md

+### Parameters
+
+Requires the `epoch` (`Epoch`) query parameter to determine which epoch will be
+considered the current epoch.


I think this should be a RequestBody with epoch and pubkeys as parameters.

peterbitfly · 2019-12-18T12:55:59Z

Thanks a lot for your efforts. Today we completed the integration of the lighthouse rpc api into our block explorer. Must of the calls work as expected, we just found 1 oddity:

For some states calling /beacon/validators/all returns Unable to read pubkey cache: PubkeyCacheIncomplete { cache_len: 0, registry_len: 16384 }

An example would be retrieving the validators with the state root of slot 192:
http://localhost:5052/beacon/validators/all?state_root=0xdf0b538dbf80d2a1d1933b80052a740e1a4f07297f83c8948650977ecf09d418

paulhauner · 2019-12-18T23:59:57Z

For some states calling /beacon/validators/all returns Unable to read pubkey cache: PubkeyCacheIncomplete { cache_len: 0, registry_len: 16384 }

Thanks @ppratscher, I know exactly what this is and will fix it.

paulhauner · 2019-12-19T00:02:09Z

All comments have been addressed :)

michaelsproul

Looks good!

Add validator index to duties response

26e7675

paulhauner added the work-in-progress label Dec 11, 2019

paulhauner self-assigned this Dec 11, 2019

paulhauner mentioned this pull request Dec 11, 2019

API additions required for blockchain explorers #694

Closed

paulhauner added 18 commits Dec 11, 2019

Add get_state method to beacon chain

51161a9

Improve /beacon/validators endpoint

686d46f

Add validators/all and validators/active endpoints

442b60f

Start refactor of HTTP docs

7bbc824

Document /beacon/heads endpoint

d6239ff

Remove some unused API endpoints

fcedd0b

Improve API docs

ae352c3

Add methods to get all validator duties

9596c18

Improve docs

707fa3d

Merge branch 'master' into bitfly

db198c9

Remove dead links

7320c8b

Make tables left-justified

378007b

Add /consensus/vote_count endpoint

aeb838d

Add /consensus/individual_votes endpoint

dc3c41e

Update formatting

02c27d8

Tidy

3b084db

Merge branch 'master' into bitfly

a6b3dd4

Add committees endpoint

bbf2acb

paulhauner added 6 commits Dec 12, 2019

Merge branch 'bitfly' of github.com:sigp/lighthouse into bitfly

c1fe02e

Strictly require 0x prefix for serde in BLS

77a43e0

Update docs to have 0x prefix

a644f1e

Fix failing tests

4b01be2

Merge branch 'master' into bitfly

9c82782

Add unfinished code

6dd2e54

Tidy, ensure all beacon endpoints smoke tested

5878a79

paulhauner added ready-for-review and removed work-in-progress labels Dec 13, 2019

paulhauner marked this pull request as ready for review Dec 13, 2019

paulhauner added this to the v0.2.0 milestone Dec 15, 2019

paulhauner removed this from the v0.2.0 milestone Dec 15, 2019

paulhauner added this to the v0.1.1 milestone Dec 15, 2019

michaelsproul self-requested a review Dec 15, 2019

michaelsproul requested changes Dec 16, 2019

View changes

pawanjay176 reviewed Dec 16, 2019

View changes

paulhauner mentioned this pull request Dec 17, 2019

Filter gossipsub message duplication #736

Merged

paulhauner mentioned this pull request Dec 17, 2019

Adds support for the step parameter in BlocksByRange request #737

Merged

michaelsproul added waiting-on-author and removed ready-for-review labels Dec 18, 2019

paulhauner added 3 commits Dec 19, 2019

Fix pubkey cache error

effd779

Address comments with docs

6c54b92

Merge branch 'bitfly' of github.com:sigp/lighthouse into bitfly

a4088f5

paulhauner added ready-for-review and removed waiting-on-author labels Dec 19, 2019

Merge branch 'master' into bitfly

cfe27aa

michaelsproul approved these changes Dec 19, 2019

View changes

michaelsproul added ready-to-merge and removed ready-for-review labels Dec 19, 2019

paulhauner merged commit 251aea6 into master Dec 19, 2019
7 checks passed

AgeManning deleted the bitfly branch Jan 3, 2020

sigp / lighthouse Public

Make API friendly to block explorers #702

Make API friendly to block explorers #702

paulhauner commented Dec 11, 2019 •

edited

paulhauner commented Dec 12, 2019

peterbitfly commented Dec 12, 2019

pscott commented Dec 15, 2019

michaelsproul left a comment

michaelsproul Dec 16, 2019

michaelsproul Dec 16, 2019

michaelsproul Dec 16, 2019

michaelsproul Dec 16, 2019

michaelsproul Dec 16, 2019

pawanjay176 Dec 16, 2019 •

edited

pawanjay176 Dec 16, 2019

peterbitfly commented Dec 18, 2019

paulhauner commented Dec 18, 2019

paulhauner commented Dec 19, 2019

michaelsproul left a comment


		### Returns

		Returns an object containing a single [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/v0.9.2/specs/core/0_beacon-chain.md#beaconblock) and it's signed root.


		## `/network/peers`

		Requests the beacon node for one `MultiAddr` for each connected peer.


		## `network/enr`

		Requests the beacon node for it's listening `ENR` address.

sigp / lighthouse Public

Make API friendly to block explorers #702

Make API friendly to block explorers #702

Conversation

paulhauner commented Dec 11, 2019 • edited

Issue Addressed

Proposed Changes

Notes

paulhauner commented Dec 12, 2019

peterbitfly commented Dec 12, 2019

pscott commented Dec 15, 2019

michaelsproul left a comment

michaelsproul Dec 16, 2019

Choose a reason for hiding this comment

michaelsproul Dec 16, 2019

Choose a reason for hiding this comment

michaelsproul Dec 16, 2019

Choose a reason for hiding this comment

michaelsproul Dec 16, 2019

Choose a reason for hiding this comment

michaelsproul Dec 16, 2019

Choose a reason for hiding this comment

pawanjay176 Dec 16, 2019 • edited

Choose a reason for hiding this comment

pawanjay176 Dec 16, 2019

Choose a reason for hiding this comment

peterbitfly commented Dec 18, 2019

paulhauner commented Dec 18, 2019

paulhauner commented Dec 19, 2019

michaelsproul left a comment

paulhauner commented Dec 11, 2019 •

edited

pawanjay176 Dec 16, 2019 •

edited