feat(flashblocks): add RLP encoding/decoding and brotli compression on flashblocks payload#269
Conversation
Cuts the per-flashblock wire payload from ~1.2 MB to ~150-250 KB under stress
by (a) RLP-encoding the EIP-7928 BlockAccessList in the upstream metadata
struct and (b) brotli-compressing every frame on both the WS and P2P pipes.
Wire/protocol changes:
- OpFlashblockPayloadMetadata.access_list becomes Option<Bytes> (RLP-encoded);
new ::new() constructor encodes from a structured BlockAccessList,
block_access_list() lazily decodes on demand
- WS pipe now carries the same Message-shaped, brotli-compressed bytes as the
P2P pipe; producer filters OpBuiltPayload from WS publish
- P2P framing switches from LinesCodec to LengthDelimitedCodec (binary-safe
for brotli output)
- Receiver auto-detects compressed vs legacy uncompressed JSON via leading-`{`
heuristic in compress::try_decompress
Encode-once relay:
- Node::run encodes Message exactly once via frame::encode and reuses the
bytes for both broadcast_message and ws_pub.publish
- BroadcastFrame { bytes, decoded } surfaces both wire bytes and decoded form
on the P2P recv path; relay handlers forward frame.bytes without re-encoding
- WsFrame mirrors this on the RPC WS recv path; persist.rs::handle_relay_flashblocks
forwards frame.bytes straight to downstream WS subscribers
Cache decode-at-insert:
- RawFlashblocksEntry adds block_access_lists: BTreeMap<u64, BlockAccessList>;
insert_flashblock decodes the RLP once, access_list_up_to is allocation-only
over the pre-decoded structures
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The flashblocks WS pipe now ships brotli-compressed `Message` envelopes as binary frames (commit 2e0cf4b), but the in-process `FlashblocksListener` still matched only `Message::Text` and parsed JSON directly, so every frame was dropped and the 5 builder flashblocks tests asserted on 0 captured flashblocks. Decode through `broadcast::frame::decode` so the listener matches the producer's wire format and works for both compressed binary and legacy uncompressed JSON. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@claude review for me |
XLayer-Bot
left a comment
XLayer-Bot
left a comment
There was a problem hiding this comment.
This PR cleanly achieves its stated goal of cutting flashblock wire size ~5-8× via brotli compression and encode-once relay semantics. The BroadcastFrame/WsFrame carrier pattern is well-designed and the auto-detect heuristic is appropriate for the backward-compatibility goal. Three issues need to be addressed before merge.
| //! (validator pipeline, persistence) read `decoded`; the relay-to-downstream | ||
| //! WS subscribers path passes `bytes` straight through to | ||
| //! [`xlayer_builder::broadcast::WebSocketPublisher::publish_raw`] without | ||
| //! re-encoding. |
There was a problem hiding this comment.
Stale rustdoc link: the module-level doc references publish_raw, but no such method exists on WebSocketPublisher. The actual method is publish (as correctly referenced in the field-level doc on line 21 of the same file). Under rustdoc -D warnings this will produce a broken-link error.
| //! re-encoding. | |
| //! [] without |
| use tokio_util::{ | ||
| codec::{FramedWrite, LinesCodec}, | ||
| codec::{FramedWrite, LengthDelimitedCodec}, | ||
| compat::FuturesAsyncReadCompatExt as _, |
There was a problem hiding this comment.
This is a hard protocol break at the framing layer. LinesCodec delimits with \n; LengthDelimitedCodec prefixes each frame with a 4-byte big-endian length header. A new-version node and an old-version node cannot exchange P2P frames at all: the legacy reader will block waiting for a newline that never arrives, and the legacy writer will emit bare JSON without a length prefix that the new reader cannot parse.
The compress::try_decompress auto-detect only handles payload-level compatibility (brotli vs. plain JSON) and does not rescue the framing mismatch. The cluster must be upgraded atomically, or a version-aware codec/handshake is needed for a rolling upgrade.
Please document the required deployment order (e.g., full-cluster stop-the-world upgrade) in the PR description or a migration note so operators don't accidentally strand a node.
There was a problem hiding this comment.
Acknowledged — this is a hard P2P framing break. Will fold the atomic-upgrade callout (sequencer + relays must be redeployed together; rolling upgrade strands nodes) into the parent PR #261 description so it's surfaced in the umbrella breaking-change notes for the stack.
| ); | ||
| if let Err(e) = stream | ||
| .send(Message::Binary(tokio_tungstenite::tungstenite::Bytes::from(payload))) | ||
| .await |
There was a problem hiding this comment.
The WS publisher now sends Message::Binary frames where it previously sent Message::Text. The internal WsFlashBlockStream receive side handles both frame types, so relay nodes are fine. However, any external WS subscriber (third-party block builders, RPC clients, monitoring tooling) that only handles text frames will silently stop receiving flashblocks after this deploy — there's no error, the connection stays open, frames just get dropped.
If there are known external consumers, they should be updated before or alongside this change. At minimum, the release notes should call out the text→binary switch as a client-breaking change.
There was a problem hiding this comment.
Acknowledged — the WS text→binary switch is a client-breaking change for any external subscriber that only handles text frames. Will fold the callout (third-party builders / RPC clients / monitoring must update to handle binary frames; failure mode is silent frame drops, no error) into the parent PR #261 description so it's surfaced in the umbrella breaking-change notes for the stack and picked up in release notes.
2 participants
Summary
Cuts the per-flashblock wire payload from ~1.2 MB to ~150–250 KB under stress by:
BlockAccessListinOpFlashblockPayloadMetadata.access_list(lazy-decoded on demand, decoded-once at cache insert)broadcast::frame::{encode,decode}single path, with auto-detect on the receive side for legacy uncompressed JSONWire / protocol changes
Message-shaped, brotli-compressed bytes as the P2P pipe (sent as binary frames);OpBuiltPayloadis filtered from WS publishLinesCodectoLengthDelimitedCodec(binary-safe for brotli output){heuristic incompress::try_decompressEncode-once relay
Node::runencodesMessageexactly once viaframe::encodeand reuses the bytes for bothbroadcast_messageandws_pub.publishBroadcastFrame { bytes, decoded }andWsFramecarry both wire bytes and decoded form on the recv path; relay handlers forwardframe.byteswithout re-encodingP2P framing is a hard break.
LinesCodec(newline-delimited) →LengthDelimitedCodec(4-byte big-endian length prefix). A new-version node and an old-version node cannot exchange P2P frames at all — the payload-level brotli auto-detect does not rescue the framing mismatch. The flashblocks cluster (sequencer + relays) must be upgraded atomically (stop-the-world); a rolling upgrade will strand nodes.WS clients: text → binary frame switch. The WS publisher now emits
Message::Binaryinstead ofMessage::Text. External WS subscribers (third-party builders, RPC clients, monitoring tooling) that only handle text frames will silently stop receiving flashblocks after this deploy — connection stays open, frames are dropped without error. Known external consumers must be updated to handle binary frames before/alongside this rollout, and this needs to be called out in the release notes.Test plan
cargo test -p xlayer-builder --lib tests::flashblockspasses (test listener updated to decode the new wire format)just check(fmt + clippy + tests)🤖 Generated with Claude Code