Consider a stateless design on the PBS module

[aimxhaisse]

When operating at scale it is convenient to have a stateless approach where you can scale the number of commit-boost instances blindly:

This allows to setup an ingress used by all beacons that spreads requests to all commit-boost instances in round-robin or more evolved forms. This has the following advantages:

• Scales well the registration calls: if you have a large infra (for example operating 100 000 validators on Holesky), you don't want a single instance of commit-boost to handle all registrations because it will be a bottleneck: relayers often return rate-limited 4XX upon large load, so you spreading from multiple IPs (instances) is needed.

• Operationally speaking it is easy to manage: you can upgrade one by one your instances with 0 downtime, canary some to qualify behavior, etc.

It has the following drawbacks on the design-side: PBS would have to be stateless, with the implication that a blinded block call for instance can land on a different instances that returned the header. This means the commit-boost instance can't know from which relayer it got the header: this is not an issue because to improve global propagation it is convenient to send the block anyway to all relayers (more than one can know about the block).

This kind of code becomes not possible with this approach: https://github.com/Commit-Boost/commit-boost-client/blob/main/crates/pbs/src/routes/submit_block.rs#L66

To me the stateless approach works very well at scale with PBS, I don't see how we could operate properly without this. However I'm unsure how it'd work for other modules.

[ltitanb]

Thanks for your input! I think the example you highlighted and the slotUUID are essentially the only points where state is needed.
We could have the following: an optional flag enables the PBS module to operate in a stateless mode. This should be quite straightforward and would disable the warning on missed payloads, but then the assumption would be that the node operator has some alternative setup to keep track of this.

This would limit some modules which do require state, which seems fine to me.

What do you think?

[aimxhaisse]

Is it a problem if the signed header is sent to all relayers? One approach (that works at scale) for blinded blocks in a stateless way:

• send the signed payload to all configured relayers, regardless of where it comes from: relayers that know about it (there can be multiple ones) will propagate it, others will just return an error,
• If you get an error from all relayers, then you can print the missed payload error.

This has the advantage of increasing propagation of the block as multiple relayers will do it.

The downside is: you could in theory submit a block via a relayer not configured for a specific key, however it's not problematic as this block was generated from a configured relayer.

[ltitanb]

To clarify, this is also the current design: we send the signed blinded block to all relays regardless of where the header came from

commit-boost-client/crates/pbs/src/mev_boost/submit_block.rs

Lines 104 to 105 in 624647a

	// we request payload to all relays, but some may have not received it
	warn!(?err, "failed to get payload (this might be ok if other relays have it)");

the state that i was mentioning above was that we keep track of the headers delivered by each relay and send an error with the list of relays that delivered the payload if the payload is failed to be delivered

commit-boost-client/crates/pbs/src/routes/submit_block.rs

Lines 51 to 66 in 624647a

	if let Some(fault_pubkeys) = state.get_relays_by_block_hash(slot, block_hash) {
	let fault_relays = state
	.relays()
	.iter()
	.filter(|relay| fault_pubkeys.contains(&relay.pubkey()))
	.map(|relay| &**relay.id)
	.collect::<Vec<_>>()
	.join(",");
	error!(%err, %block_hash, fault_relays, "CRITICAL: no payload received from relays");
	state.publish_event(BuilderEvent::MissedPayload {
	block_hash,
	relays: fault_relays,
	});
	} else {
	error!(?err, %block_hash, "CRITICAL: no payload delivered and no relay for block hash. Was getHeader even called?");

but this can probably be handled separately if running in "stateless" mode