Add block v3 endpoint. #339

mcdee · 2023-07-16T10:13:38Z

As per #309 this adds a version 3 block endpoint that consolidates the current blinded and unblinded endpoints. Features of this endpoint:

a single endpoint to fetch blocks regardless of type
allows the beacon node to decide (with help from the EL) if it should use MEV relays for execution payload
provides the returned block's payload to the proposer
metadata allows for simpler handling of decoding the returned data

rkapka · 2023-07-17T12:55:06Z

beacon-node-oapi.yaml

@@ -421,3 +424,13 @@ components:
      required: true
      schema:
        $ref: '#/components/schemas/ConsensusVersion'
+    Eth-Blinded-Payload:
+      description: Required in response so client can deserialize returned json or ssz data to the correct object.


Suggested change

description: Required in response so client can deserialize returned json or ssz data to the correct object.

description: Required in response so client can deserialize returned JSON or SSZ data to the correct object.

The existing Eth-Consensus-Version uses lower-case so I stuck with it for consistency.

dapplion · 2023-07-20T13:14:01Z

apis/validator/block.v3.yaml

+        Eth-Consensus-Version:
+          $ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Consensus-Version'
+        Eth-Blinded-Payload:
+          $ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Blinded-Payload'
+        Eth-Payload-Value:
+          $ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Payload-Value'


Headers are required for SSZ returns. Should the added two headers be also present as JSON values to the application/json response type? Otherwise, why include version as property but not the payload value.

Yeah, wasn't sure about that. In the past we added the version to the JSON to allow for decoding, but it has pretty much been replaced with the header values now (as you don't need to do any mucking around with parsing streams to obtain the values from the headers). I'm not averse to putting these new values in the JSON metadata, but not sure if they add significant value any more.

do we do something controversial and just return the json object not wrapped in a 'data' and have it just the json object or the ssz object determined by header?

do we do something controversial and just return the json object not wrapped in a 'data' and have it just the json object or the ssz object determined by header?

It would definitely break the standard from other routes and prevent allowing to add more fields in the future. I would mirror the header values in the JSON type for overall consistency. It's easier to parse a JSON reply than headers for client side tooling that deals with JSON only

@dapplion yeah I'm thinking the same. If the metadata fields are in the JSON then it will allow parsing of the JSON separate from the HTTP response (e.g. if the JSON is saved and parsed later). I'll update accordingly.

Updated. One question that rises from this is the naming of the fields: do we want to explicitly state which values are related to the execution payload, i.e.

blinded-execution-payload rather than blinded-payload

execution-payload-value rather than payload-value

(and equivalent change for the header fields). Obviously this makes them a little more verbose, but is safer if we ever end up with other payloads in the beacon block. Thoughts?

if we went that way, i'd probably suggest something like

execution-payload-value execution-payload-blinded

i don't mind either way on whether we expand it...

Have updated to use these names, agree that they are a bit clearer. I think that this looks like the best way forward;
@dapplion @rolfyone agree/disagree?

dapplion

Looks good! Minor nit on mirroring header values

apis/validator/block.v3.yaml

rolfyone

LGTM will just make sure others are happy...

dapplion

Looks good!

g11tech

~~can't find produceBlindedBlockV3 in PR, is this a mistake or intentional~~
ahh realized this is replacement for both, so then can be add another query param like blockSource ("builder", "execution" or "any")
can be also add shouldOverrideBuilder flag in response so that validator driven architectures of making the choice (like lodestar validator or may be vouch) can consume this info
can we also add the blockSource ("builder" or "execution") to the response apart from the query to reflect the selection here (especially in case of any), also gives a way to figure out that if the publishBlock call is binded to a particular beacon in case of blinded response

mcdee · 2023-07-25T12:36:21Z

can't find produceBlindedBlockV3 in PR, is this a mistake or intentional

Not sure to what this refers. There is a produceBlindedBlock operation, but that is specifically for producing blinded blocks alone. The operation ID of this call is produceBlockV3.

can be also add shouldOverrideBuilder flag in response so that validator driven architectures of making the choice (like lodestar validator or may be vouch) can consume this info

Beacon nodes will by default return the highest value block from either the relay(s) or the local payload. However, if the beacon node's circuit breaker is triggered or the execution client specifies that the local payload should be used then the local payload will be returned regardless of value. Does knowing why the full payload was returned (higher value / consensus circuit breaker / execution shouldOverrideBuilder) have any value here?

can we also add the blockSource ("builder" or "execution") to the response

This can be inferred from the returned payload:

blinded => comes from the builder
unblinded => comes from the execution node

g11tech · 2023-07-25T12:47:16Z

lodestar validator and potentially other validator driven approaches use the info to decide what to use in validator and don't decide between execution and builder in beacon node, so would be good to flow data
i. in previous discussions we have (it has been some time now) we sort of agreed to let these two kind of approaches live side by side and this extra info will help that and is harmless and quite small to pass around
ii. thats why a similar blinded version is required (although we can transition to this but am inclined towards validator driven approach atleast for implementation diversity)
even execution can return blinded which will be important for the big payloads specially in deneb, so the inference should better be explicit

imo, this only needs to pass certain more flags around which are very small in data size and doesn't require any extra implementation work.

mcdee · 2023-07-25T12:57:08Z

I think that although the idea of having validator-controlled and beacon node-controlled selection of blocks as separate streams was nice in theory, in reality the beacon node is the entity that decides from where to fetch the execution payload. The validator client ultimately has no choice in the matter; it can only use what the beacon node returns, regardless of what it asks for. As such, I'm not sure what the purpose of the suggested flags will serve (specifically, in which situation would the validator client not ask for "any" as their block, given they want to maximize the value of the execution payload).
as @arnetheduck says in Consolodating blinded/unblinded flow #309 (comment) we want the beacon node to be stateless as much as we can manage, which would require returning full blocks whenever possible

g11tech · 2023-07-25T13:32:49Z

I think that although the idea of having validator-controlled and beacon node-controlled selection of blocks as separate streams was nice in theory, in reality the beacon node is the entity that decides from where to fetch the execution payload. The validator client ultimately has no choice in the matter; it can only use what the beacon node returns, regardless of what it asks for. As such, I'm not sure what the purpose of the suggested flags will serve (specifically, in which situation would the validator client not ask for "any" as their block, given they want to maximize the value of the execution payload).

as @arnetheduck says in Consolodating blinded/unblinded flow #309 (comment) we want the beacon node to be stateless as much as we can manage, which would require returning full blocks whenever possible

shouldn't these be left to implementation and support the implementation types. again lodestar can transition to this paradigm but should be nice to have and support implementation diversity.

mcdee · 2023-07-25T13:43:59Z

Which flags are you after here? The one that was under discussion was to tell the beacon node to not override use of a relay payload with a local payload (when it had been informed to either from its own circuit breaker or from the execution API shouldOverrideBuilder flag), but it seems to me that it makes more sense to make this a configuration option for the beacon node itself rather than a parameter passed each time a block is requested.

g11tech · 2023-07-26T07:02:01Z

Which flags are you after here? The one that was under discussion was to tell the beacon node to not override use of a relay payload with a local payload (when it had been informed to either from its own circuit breaker or from the execution API shouldOverrideBuilder flag), but it seems to me that it makes more sense to make this a configuration option for the beacon node itself rather than a parameter passed each time a block is requested.

lodestar current impl needs all the flags i mentioned before, but may be we will change our impl to push the block selection to beacon node.

but then atleast blockSource should be there in the response so that if the beacon node chooses blinded response even for executon block, (configured for e.g. via a startup flag for beacon node) then validator knows that the publish call is bound to the same beacon node. For lodestar this matters as we have sometimes seen latency for block communication between beacon and validator and have even lead to some miss scenaros

arnetheduck · 2023-09-15T12:13:38Z

beacon-node-oapi.yaml

+      required: true
+      schema:
+        type: boolean
+    Eth-Execution-Payload-Value:


Without the consensus value, there's no (reasonable) way for a VC to compare blocks for profit - should we add this here?

Does Nimbus already compute this value during block proposal? Lighthouse sort of does, but approximates slightly (if there are slashable attestations we don't keep track of the rewards from both).

It would make VC block choosing more fair, as the heuristics Vouch uses currently seem to favour Prysm despite Prysm's blocks being less profitable than Lighthouse and Nimbus on average (according to my blockprint nodes, public dash coming soon).

could this be a new PR? this one is closed.

Does Nimbus already compute this value during block proposal?

We do it for offline analysis for historical blocks (to pick out more detail than is needed for state transitions) but making it "online" would probably not be too hard - indeed, the VC currently has to do a lot of guesswork and it will likely always be wrong / skewed - further exacerbating the problem is that in that guesswork, it's not working with eth-denominated data usually but rather "fresh attestation count" and similar proxies which prevents the value from being compared with the execution profit (ie low cl+high el vs low cl+high el score)

could this be a new PR? this one is closed.

#358

mcdee added 2 commits July 16, 2023 11:05

Add block v3 endpoint.

07d9752

Deprecate blinded block request.

7d11c25

rkapka reviewed Jul 17, 2023

View reviewed changes

dapplion reviewed Jul 20, 2023

View reviewed changes

james-prysm mentioned this pull request Jul 17, 2023

EIP4844 Feature Completion Checklist prysmaticlabs/prysm#12248

Closed

40 tasks

timbeiko mentioned this pull request Jul 20, 2023

EIP-4844 Implementers' Call #27 ethereum/pm#835

Closed

mcdee added 2 commits July 21, 2023 15:28

Add JSON metadata.

8766d92

Clearer metadata names.

d3b9cce

dapplion reviewed Jul 24, 2023

View reviewed changes

apis/validator/block.v3.yaml Outdated Show resolved Hide resolved

Use standard name for metadata.

fca70b2

rolfyone approved these changes Jul 25, 2023

View reviewed changes

dapplion approved these changes Jul 25, 2023

View reviewed changes

g11tech suggested changes Jul 25, 2023

View reviewed changes

Merge branch 'master' into block-v3

6a0755e

mehdi-aouadi mentioned this pull request Jul 26, 2023

Add block v3 endpoint Consensys/teku#7390

Closed

7 tasks

rolfyone merged commit 20980bf into ethereum:master Jul 28, 2023
2 checks passed

jimmygchen mentioned this pull request Aug 9, 2023

Add block v3 endpoint sigp/lighthouse#4582

Closed

eserilev mentioned this pull request Aug 9, 2023

[WIP] Add block v3 endpoint sigp/lighthouse#4594

Closed

james-prysm mentioned this pull request Aug 10, 2023

Deneb produce blockv3 prysmaticlabs/prysm#12708

Merged

2 tasks

arnetheduck reviewed Sep 15, 2023

View reviewed changes

Falehfale mentioned this pull request Sep 28, 2023

F #364

Closed

jimmygchen mentioned this pull request Oct 6, 2023

Use validator/blocks v3 endpoint in the VC sigp/lighthouse#4812

Closed

tersec mentioned this pull request Dec 3, 2023

Update to current (deprecated, but) version of produceBlindedBlock status-im/nimbus-eth2#5639

Merged

dapplion mentioned this pull request Dec 8, 2023

Consolodating blinded/unblinded flow #309

Closed

pinebit mentioned this pull request Dec 20, 2023

Add support to produceBlockV3 attestantio/go-eth2-client#97

Closed

chenrui333 mentioned this pull request Jan 28, 2024

lighthouse 4.6.0 Homebrew/homebrew-core#161115

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add block v3 endpoint. #339

Add block v3 endpoint. #339

mcdee commented Jul 16, 2023

rkapka Jul 17, 2023

mcdee Jul 17, 2023

dapplion Jul 20, 2023

mcdee Jul 20, 2023

rolfyone Jul 21, 2023

dapplion Jul 21, 2023

mcdee Jul 21, 2023

mcdee Jul 21, 2023

rolfyone Jul 22, 2023

mcdee Jul 22, 2023

dapplion left a comment •

edited

Loading

rolfyone left a comment

dapplion left a comment

g11tech left a comment •

edited

Loading

mcdee commented Jul 25, 2023

g11tech commented Jul 25, 2023 •

edited

Loading

mcdee commented Jul 25, 2023

g11tech commented Jul 25, 2023

mcdee commented Jul 25, 2023

g11tech commented Jul 26, 2023 •

edited

Loading

arnetheduck Sep 15, 2023

michaelsproul Sep 15, 2023

rolfyone Sep 18, 2023

arnetheduck Sep 20, 2023

arnetheduck Sep 20, 2023

	description: Required in response so client can deserialize returned json or ssz data to the correct object.
	description: Required in response so client can deserialize returned JSON or SSZ data to the correct object.

Add block v3 endpoint. #339

Add block v3 endpoint. #339

Conversation

mcdee commented Jul 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion left a comment • edited Loading

Choose a reason for hiding this comment

rolfyone left a comment

Choose a reason for hiding this comment

dapplion left a comment

Choose a reason for hiding this comment

g11tech left a comment • edited Loading

Choose a reason for hiding this comment

mcdee commented Jul 25, 2023

g11tech commented Jul 25, 2023 • edited Loading

mcdee commented Jul 25, 2023

g11tech commented Jul 25, 2023

mcdee commented Jul 25, 2023

g11tech commented Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion left a comment •

edited

Loading

g11tech left a comment •

edited

Loading

g11tech commented Jul 25, 2023 •

edited

Loading

g11tech commented Jul 26, 2023 •

edited

Loading