Add lightclient routes #181

dapplion · 2021-12-03T12:41:06Z

Adds minimal set of lightclient routes to super a server-based trust-less censorable lightclient.

Sample usage flow for these APIs:

Gather weak subjectivity checkpoint root from some trusted source
Call /eth/v1/lightclient/snapshot/{block_root} to get an initial SyncCommittee
Sync from initial SyncCommittee's sync period to clock sync period with /eth/v1/lightclient/committee_updates
Subscribe to head update events
4.1. Subscribe to /eth/v1/events?topics=lightclient_head_update
4.2. If SSE are not available or to get the first head event faster, poll /eth/v1/lightclient/head_update
Once close to the end of the current clock sync period, call /eth/v1/lightclient/committee_updates

For a sample implementation of this routes see:

Routes backend incorporated into a beacon node: https://github.com/ChainSafe/lodestar/blob/master/packages/lodestar/src/chain/lightClient/index.ts#L159
Routes front-end in a stand-alone static application: https://github.com/ChainSafe/eth2-light-client-demo/blob/master/src/App.tsx

Live demo: https://lodestar-light-client-demo.netlify.app

mcdee · 2021-12-04T09:40:45Z

apis/eventstream/index.yaml

@@ -73,6 +74,17 @@ get:
              value: |
                event: contribution_and_proof
                data: {"message": {"aggregator_index": "997", "contribution": {"slot": "168097", "beacon_block_root": "0x56f1fd4262c08fa81e27621c370e187e621a67fc80fe42340b07519f84b42ea1", "subcommittee_index": "0", "aggregation_bits": "0xffffffffffffffffffffffffffffffff", "signature": "0x85ab9018e14963026476fdf784cc674da144b3dbdb47516185438768774f077d882087b90ad642469902e782a8b43eed0cfc1b862aa9a473b54c98d860424a702297b4b648f3f30bdaae8a8b7627d10d04cb96a2cc8376af3e54a9aa0c8145e3"}, "selection_proof": "0x87c305f04bfe5db27c2b19fc23e00d7ac496ec7d3e759cbfdd1035cb8cf6caaa17a36a95a08ba78c282725e7b66a76820ca4eb333822bd399ceeb9807a0f2926c67ce67cfe06a0b0006838203b493505a8457eb79913ce1a3bcd1cc8e4ef30ed"}, "signature": "0xac118511474a94f857300b315c50585c32a713e4452e26a6bb98cdb619936370f126ed3b6bb64469259ee92e69791d9e12d324ce6fd90081680ce72f39d85d50b0ff977260a8667465e613362c6d6e6e745e1f9323ec1d6f16041c4e358839ac"}
+            lightclient_head_update:


I don't see a need for _update here; the event is, by definition, an update.

Good point, tho I'm unsure if lightclient_head can also be confusing since you can get multiple events for the same head which is not the same behavior as the head event.

I think it was called lightlcient_head_update because the data structure emitted is called a LightClientHeadUpdate somewhere.

Similar to the LightClientUpdate in the specs.

Not saying its a great name, just saying..

apis/eventstream/index.yaml

mcdee · 2021-12-04T09:51:49Z

apis/eventstream/index.yaml

+                The node has received a new or better lightclient head update. `header` is the block header signed by `sync_aggregate`.
+                Updates may be collected via
+                - Next block's syncAggregate field
+                - Aggregating messages from gossip topic `sync_committee_contribution_and_proof`


Is this considered safe, if this information is not yet in a (voted upon) block? Is there any way of knowing the level of finality of the provided head?

Is this considered safe, if this information is not yet in a (voted upon) block?

Yes, adding the SyncAggregate is only useful to give on-chain incentives to sync committee members. Gathering the data from gossip topics is identical.

Is there any way of knowing the level of finality of the provided head?

Not really, the main data points lightclients can rely on are:

participation of this SyncAggregate

if the block is finalized or not

apis/lightclient/committee_updates.yaml

apis/lightclient/head_update.yaml

mcdee · 2021-12-04T09:57:01Z

apis/lightclient/head_update.yaml

+  summary: Get latest best head update
+  description: |
+    Returns the latest best head update available. Clients should use the SSE type `lightclient_head_update`
+    unless to get the very first head update after syncing, or if SSE are not supported by the server.


Events are part of the spec so the latter part is not a relevant comment.

the latter part is not a relevant comment.

Do you refer to the part where SSE may not be supported by the server? At least Lodestar allows to disable the events namespace individually. Also some reverse proxies may not support SSE depending on their buffering configuration.

mcdee · 2021-12-04T09:59:57Z

apis/lightclient/proof.yaml

+    "200":
+      description: Success
+      content:
+        application/octet-stream:


There should be a JSON response here as well. This will also give a location to explain the structure of the multiproof.

The JSON representation of this format is widely inefficient. Since JSON is the default I'll argue into only offering application/octet-stream to force consumers to use the efficient path

apis/lightclient/snapshot.yaml

mcdee · 2021-12-04T10:03:57Z

beacon-node-oapi.yaml

@@ -96,6 +98,15 @@ paths:
  /eth/v1/debug/beacon/heads:
    $ref: './apis/debug/heads.yaml'

+  /eth/v1/lightclient/committee_updates:


The API has historically defined committee as a beacon committee, and sync_committee as a sync committee, so I believe you want the latter here.

Even if it's scoped in the lightclient namespace is there a risk of confusion?

I support using sync_committee for clarity here.

arnetheduck · 2021-12-04T10:42:50Z

This is great as a draft, but believe it's good to settle on a p2p protocol before enshrining / merging it such that implementations don't have to maintain two views of the same data that are slightly different - this work has started in ethereum/consensus-specs#2267 and should be completed first - we are in the process of implementing it and will be providing updates soon.

The long-term idea here would be that beacon nodes serve light client data via p2p just like they serve blocks which enables a very fast consensus sync in general and where serving the data would be part of the standard feature set offered by clients via P2P (to later be mirrored in the REST API).

mcdee · 2021-12-04T11:21:20Z

A few generic questions:

should /eth/v1/lightclient/snapshot/ only take a block root, or should it allow the more generic block ID? Rationale behind this would be to allow simple retrieval of the snapshot from a node which is considered trusted.
how should a client obtain the period(s) in step 3 given the data obtained so far? Seems like they need the spec to be able to go from slot to sync committee period, but should this be trusted? Would it make sense to use slots here for the from and to parameters, and have the beacon node translate them to sync committee periods?
for 5) how does the client know when the sync period is coming up for a change? Perhaps a slots_until_next_sync_committee_period or similar should be added to the snapshot information?

dapplion · 2021-12-05T09:18:34Z

RE @mcdee

should /eth/v1/lightclient/snapshot/ only take a block root, or should it allow the more generic block ID? Rationale behind this would be to allow simple retrieval of the snapshot from a node which is considered trusted.

Good point! Tho at least one block ID value is not suitable "genesis". Also if you use head you will have to guess the block_root if you want to validate the merkle proof. If you want a SyncCommittee trusting the node you can just query /eth/v1/beacon/states/{state_id}/sync_committees and /eth/v1/beacon/headers/{block_id}

how should a client obtain the period(s) in step 3 given the data obtained so far? Seems like they need the spec to be able to go from slot to sync committee period, but should this be trusted? Would it make sense to use slots here for the from and to parameters, and have the beacon node translate them to sync committee periods?

The config, genesis_time and genesis_validators_root of the network should be known ahead of time by the lightclient consumer. Then you can compute sync periods from slots. Using sync periods allows to put a cache layer in front of the node to speed up responses. If using slots, the same response can be requested with way more different requests.

for 5) how does the client know when the sync period is coming up for a change? Perhaps a slots_until_next_sync_committee_period or similar should be added to the snapshot information?

This can be easily computed knowing the config, genesis_time and genesis_validators_root of the network

dapplion · 2021-12-05T09:25:05Z

RE: @arnetheduck

This is great as a draft, but believe it's good to settle on a p2p protocol before enshrining / merging it such that implementations don't have to maintain two views of the same data that are slightly different - this work has started in ethereum/consensus-specs#2267 and should be completed first - we are in the process of implementing it and will be providing updates soon.

Thanks! Tho I believe there's value in iterating both publicly at the same time. Lodestar will tackle p2p next so happy to give feedback on ethereum/consensus-specs#2267 then

holgerd77 · 2022-04-20T08:57:21Z

Will cc myself and @ryanio in here.

etan-status · 2022-04-21T07:54:18Z

Feedback from discussions at Devconnect AMS:

See libp2p PR discussion: ethereum/consensus-specs#2802 (comment)

Align data types with libp2p once there is agreement there. The light client should receive the same objects regardless of the underlying transport.
- Exception being that SSE does not necessarily require the selection proof that is being discussed (similar to other events types)
Add route for fetching latest light client update to enable use case of retrieving latest finalized block root
- light_client/updates/latest
Align naming with best practices and make it consistent across transports
- Data types: Same as libp2p
  - LightClientHeaderUpdate -> LightClientOptimisticUpdate
  - LightClientUpdate stays the same
  - LightClientSnapshotWithProof -> LightClientBootstrap
- HTTP GET:
  - /lightclient/committee_updates?from={from}&to={to} -> light_client/updates?from={from}&to={to} (or use start_period / count to match libp2p?)
  - lightclient/head_update/ -> light_client/optimistic_update
  - lightclient/snapshot/{block_root} -> light_client/bootstrap/{block_root}
- HTTP Server-Sent Event:
  - lightclient_head_update -> light_client_optimistic_update

etan-status · 2022-05-10T09:36:16Z

Would also change from /eth2/v1/light_client to /eth2/v1/beacon/light_client/...as this data is about the beacon chain. It is also how the libp2p reqs are currently prefixed in proposal (at the very least, it should be consistent). On libp2p there is also the relation to the Status message which is another indicator that LC stuff should be under beacon umbrella.

Adopts the light client data REST API used by Lodestar as defined in ethereum/beacon-APIs#181 with a v0 suffix. Requests: - `/eth/v0/light_client/bootstrap/{block_root}` - `/eth/v0/light_client/updates?from={from}&to={to}` - `/eth/v0/light_client/finality_update` - `/eth/v0/light_client/optimistic_update` HTTP Server-Sent Events (SSE): - `light_client_finality_update_v0` - `light_client_optimistic_update_v0` More work is needed to adopt the proofs endpoint, it is not included.

Adopts the light client data REST API used by Lodestar as defined in ethereum/beacon-APIs#181 with a v0 suffix. Requests: - `/eth/v0/light_client/bootstrap/{block_root}` - `/eth/v0/light_client/updates?start_period={start}&count={count}` - `/eth/v0/light_client/finality_update` - `/eth/v0/light_client/optimistic_update` HTTP Server-Sent Events (SSE): - `light_client_finality_update_v0` - `light_client_optimistic_update_v0` More work is needed to adopt the proofs endpoint, it is not included.

Adopts the light client data REST API used by Lodestar as defined in ethereum/beacon-APIs#181 with a v0 suffix. Requests: - `/eth/v0/beacon/light_client/bootstrap/{block_root}` - `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}` - `/eth/v0/beacon/light_client/finality_update` - `/eth/v0/beacon/light_client/optimistic_update` HTTP Server-Sent Events (SSE): - `light_client_finality_update_v0` - `light_client_optimistic_update_v0` More work is needed to adopt the proofs endpoint, it is not included.

etan-status · 2022-06-18T16:11:26Z

Adopted with a v0 suffix in Nimbus (without proofs endpoint)

Requests:

/eth/v0/beacon/light_client/bootstrap/{block_root}
/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}
/eth/v0/beacon/light_client/finality_update
/eth/v0/beacon/light_client/optimistic_update

HTTP Server-Sent Events (SSE):

light_client_finality_update_v0
light_client_optimistic_update_v0

status-im/nimbus-eth2#3775

* adopt LC REST API with v0 suffix (without proofs) Adopts the light client data REST API used by Lodestar as defined in ethereum/beacon-APIs#181 with a v0 suffix. Requests: - `/eth/v0/beacon/light_client/bootstrap/{block_root}` - `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}` - `/eth/v0/beacon/light_client/finality_update` - `/eth/v0/beacon/light_client/optimistic_update` HTTP Server-Sent Events (SSE): - `light_client_finality_update_v0` - `light_client_optimistic_update_v0` More work is needed to adopt the proofs endpoint, it is not included. * initialize event queues * register event topics

zah · 2022-07-02T10:43:33Z

It would be useful if all requests support returning SSZ data if the Accept: application/octet-stream header is provided. To make this possible, we need to specify an upper limit to the number of results of the getCommitteeUpdates request (in order to encode the result as List[LightClientUpdate, Limit] in SSZ).

etan-status · 2022-07-02T14:51:00Z

The format is not that simple, as each element could be from different forks. The list does not solely contain altair.LightClientUpdate but may contain different updates in the future.

On Gossipsub, this is solved by prepending the fork digest to each item (similar encoding as a SSZ union, but not processed as a SSZ union as some legacy endpoints also support having an empty fork digest to indicate phase0).

Technically, you could also peek into the first 8 bytes of the SSZ response to determine attested_header.slot and infer the fork digest from there. A more general approach would be desirable though, as the 4 bytes do not add meaningful overhead.

Note that SSZ unions are not well-supported in all SSZ encoding libraries. (or only pseudo-supported with a type of 0).

dapplion · 2022-07-05T05:35:15Z

It would be useful if all requests support returning SSZ data if the Accept: application/octet-stream header is provided. To make this possible, we need to specify an upper limit to the number of results of the getCommitteeUpdates request (in order to encode the result as List[LightClientUpdate, Limit] in SSZ).

@etan-status something like this would be valuable for the beacon -> validator API too. Some previous discussion on how to handle fork versioning

V2 API calls and SSZ encoded response. #167

dapplion · 2022-09-15T13:15:41Z

After discussing with @etan-status its best to separate the proof route into a different namespace TBD

@dapplion

Introduces additional REST endpoints for light client sync, compatible with the libp2p specification defined in `ethereum/consensus-specs`: https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.3/specs/altair/light-client/p2p-interface.md Endpoints are defined to be fork-aware to allow potential optimizations such as including EL block hash into `BeaconBlockHeader` in the future. The `getLightClientUpdatesByRange` endpoint is a new kind of endpoint, as it introduces the concept of list responses. Each list item could potentially be of a different fork. Likewise, for the events stream, the fork may update between events. Both the list response and the event are additionall wrapped with a per-item `version` indicator in JSON. For SSZ, the list items are encoded as length/version/payload triples. Note that the events endpoint does not support the SSZ format. This builds on prior work from: - @dapplion at ethereum#181 The proof endpoint in that PR is out-of-scope here, but could be useful as part of the debug API as an `eth_getProof` equivalent for CL data.

@dapplion

Introduces additional REST endpoints for light client sync, compatible with the libp2p specification defined in `ethereum/consensus-specs`: https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.3/specs/altair/light-client/p2p-interface.md Endpoints are defined to be fork-aware to allow potential optimizations such as including EL block hash into `BeaconBlockHeader` in the future. The `getLightClientUpdatesByRange` endpoint is a new kind of endpoint, as it introduces the concept of list responses. Each list item could potentially be of a different fork. Likewise, for the events stream, the fork may update between events. Both the list response and the event are additionall wrapped with a per-item `version` indicator in JSON. For SSZ, the list items are encoded as length/version/payload triples. Note that the events endpoint does not support the SSZ format. This builds on prior work from: - @dapplion at ethereum#181 The proof endpoint in that PR is out-of-scope here, but could be useful as part of the debug API as an `eth_getProof` equivalent for CL data.

@dapplion

Introduces additional REST endpoints for light client sync, compatible with the libp2p specification defined in `ethereum/consensus-specs`: https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.3/specs/altair/light-client/p2p-interface.md Endpoints are defined to be fork-aware to allow potential optimizations such as including EL block hash into `BeaconBlockHeader` in the future. The `getLightClientUpdatesByRange` endpoint is a new kind of endpoint, as it introduces the concept of list responses. Each list item could potentially be of a different fork. Likewise, for the events stream, the fork may update between events. Both the list response and the event are additionall wrapped with a per-item `version` indicator in JSON. For SSZ, the list items are encoded as length/version/payload triples. Note that the events endpoint does not support the SSZ format. This builds on prior work from: - @dapplion at ethereum#181 The proof endpoint in that PR is out-of-scope here, but could be useful as part of the debug API as an `eth_getProof` equivalent for CL data.

dapplion · 2022-10-04T15:02:29Z

Closing for Define endpoints for light client sync #247 which is a superset of this PR's functionality

Proofs endpoint should be tackled separately

@dapplion

Introduces additional REST endpoints for light client sync, compatible with the libp2p specification defined in `ethereum/consensus-specs`: https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.3/specs/altair/light-client/p2p-interface.md Endpoints are defined to be fork-aware to allow potential optimizations such as including EL block hash into `BeaconBlockHeader` in the future. The `getLightClientUpdatesByRange` endpoint is a new kind of endpoint, as it introduces the concept of list responses. Each list item could potentially be of a different fork. Likewise, for the events stream, the fork may update between events. Both the list response and the event are additionall wrapped with a per-item `version` indicator in JSON. For SSZ, the list items are encoded as length/version/payload triples. Note that the events endpoint does not support the SSZ format. This builds on prior work from: - @dapplion at ethereum/beacon-APIs#181 The proof endpoint in that PR is out-of-scope here, but could be useful as part of the debug API as an `eth_getProof` equivalent for CL data.

Add lightclient routes

8d1187f

mcdee suggested changes Dec 4, 2021

View reviewed changes

dapplion added 5 commits December 5, 2021 09:52

Fix typo

3e36a24

Don't use sse term

3993376

Fix block_root param type

e751124

Clarify lightclient_head_update description

be5ed0f

Link multiproof format

c160d12

philknows mentioned this pull request Dec 13, 2021

Light Client Planning Tracker ChainSafe/lodestar#3469

Closed

5 tasks

philknows mentioned this pull request Jan 7, 2022

Light client API standardization ChainSafe/lodestar#3079

Closed

g11tech mentioned this pull request Jan 14, 2022

Update beacon-apis lightclient PR #181 with spec 1.1.7 changes ChainSafe/lodestar#3619

Closed

g11tech added 3 commits January 14, 2022 18:31

light client structs alignment with CL spec 1.1.7

f0946e3

fix SyncCommittee ref

f9c40fc

rename syncCommitteeAggregate to syncAggregate as per cl spec 1.1.9

4cc1f0a

wemeetagain mentioned this pull request Apr 26, 2022

Align light client API ChainSafe/lodestar#3952

Closed

philknows mentioned this pull request Jun 11, 2022

Lodestar Roadmap and Vision (Updated Apr 2023) 🌟 ChainSafe/lodestar#3373

Open

dadepo mentioned this pull request Jun 16, 2022

Align light client related endpoints ChainSafe/lodestar#4168

Merged

etan-status mentioned this pull request Jun 18, 2022

adopt LC REST API with v0 suffix (without proofs) status-im/nimbus-eth2#3775

Merged

zah mentioned this pull request Jun 30, 2022

Develop an initial relay component fetching light client updates from a running beacon node metacraft-labs/DendrETH#10

Closed

etan-status mentioned this pull request Jul 2, 2022

Implement SSZ responses in the light client REST APIs status-im/nimbus-eth2#3836

Open

dadepo mentioned this pull request Aug 1, 2022

Prefix the light client path with beacon ChainSafe/lodestar#4354

Closed

dadepo mentioned this pull request Aug 15, 2022

prefix light client paths with beacon ChainSafe/lodestar#4413

Merged

Remove proof endpoint

818e21b

dapplion dismissed a stale review via 818e21b September 15, 2022 13:15

etan-status mentioned this pull request Sep 22, 2022

Define endpoints for light client sync #247

Merged

dapplion closed this Oct 4, 2022

lerajk mentioned this pull request Oct 7, 2022

Lodestar Light Client Research ChainSafe/forest#2018

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lightclient routes #181

Add lightclient routes #181

dapplion commented Dec 3, 2021

mcdee Dec 4, 2021

dapplion Dec 5, 2021

wemeetagain Dec 6, 2021

mcdee Dec 4, 2021

dapplion Dec 5, 2021

mcdee Dec 4, 2021

dapplion Dec 5, 2021

mcdee Dec 4, 2021

dapplion Dec 5, 2021

mcdee Dec 4, 2021

dapplion Dec 5, 2021

claravanstaden Mar 24, 2022

arnetheduck commented Dec 4, 2021

mcdee commented Dec 4, 2021

dapplion commented Dec 5, 2021 •

edited

dapplion commented Dec 5, 2021

holgerd77 commented Apr 20, 2022

etan-status commented Apr 21, 2022 •

edited

etan-status commented May 10, 2022 •

edited

etan-status commented Jun 18, 2022

zah commented Jul 2, 2022

etan-status commented Jul 2, 2022 •

edited

dapplion commented Jul 5, 2022 •

edited

dapplion commented Sep 15, 2022

dapplion commented Oct 4, 2022

Add lightclient routes #181

Add lightclient routes #181

Conversation

dapplion commented Dec 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnetheduck commented Dec 4, 2021

mcdee commented Dec 4, 2021

dapplion commented Dec 5, 2021 • edited

dapplion commented Dec 5, 2021

holgerd77 commented Apr 20, 2022

etan-status commented Apr 21, 2022 • edited

etan-status commented May 10, 2022 • edited

etan-status commented Jun 18, 2022

zah commented Jul 2, 2022

etan-status commented Jul 2, 2022 • edited

dapplion commented Jul 5, 2022 • edited

dapplion commented Sep 15, 2022

dapplion commented Oct 4, 2022

dapplion commented Dec 5, 2021 •

edited

etan-status commented Apr 21, 2022 •

edited

etan-status commented May 10, 2022 •

edited

etan-status commented Jul 2, 2022 •

edited

dapplion commented Jul 5, 2022 •

edited