v3 websocket protocol by joshua-spacetime · Pull Request #4761 · clockworklabs/SpacetimeDB

joshua-spacetime · 2026-04-08T21:45:52Z

Description of Changes

The v3 WebSocket API adds a thin transport layer around the existing v2 message schema so that multiple logical ClientMessages can be sent in a single WebSocket frame.

The motivation is throughput. In v2, each logical client message requires its own WebSocket frame, which adds per-frame overhead in the client runtime, server framing/compression path, and network stack. High-throughput clients naturally issue bursts of requests, and batching those requests into a single frame materially reduces that overhead while preserving the existing logical message model.

v3 keeps the v2 message schema intact and treats batching as a transport concern rather than a semantic protocol change. This lets the server support both protocols cleanly:

v2 remains unchanged for existing clients
v3 allows new clients to batch logical messages without changing reducer/procedure semantics
inner messages are still processed in order

On the server side, this PR adds:

v3.bsatn.spacetimedb protocol support
ClientFrame / ServerFrame transport envelopes
decoding of inbound batched client frames into ordered v2 logical messages
v3 outbound framing on the server side

API and ABI breaking changes

None. v2 clients continue to work unchanged.

Expected complexity level and risk

2

Testing

Testing will be included in the patches that update the sdk bindings

Centril

My main comment here is that I think Client/ServerFrame don't actually need to exist and could just be Bytes which would also be more efficient and avoid some temporary allocations introduced here.

Centril · 2026-04-10T06:34:53Z

crates/client-api/src/routes/subscribe.rs

    let mut in_use_bufs: Vec<ScopeGuard<InUseSerializeBuffer, _>> = Vec::with_capacity(BUF_POOL_CAPACITY);

-    while let Some(message) = messages.recv().await {
+    'send: while let Some(message) = messages.recv().await {


Is this just for added readability? (I don't mind it.)

Centril · 2026-04-10T06:40:12Z

crates/client-api/src/routes/subscribe.rs

+/// Reports encode metrics for an already-encoded message and forwards all of
+/// its frames to the websocket send task.
+fn ws_forward_frames<I>(
+    metrics: &SendMetrics,
+    outgoing_frames: &mpsc::UnboundedSender<Frame>,
+    workload: Option<WorkloadType>,
+    num_rows: Option<usize>,
+    encoded: (EncodeMetrics, InUseSerializeBuffer, I),
+) -> Result<InUseSerializeBuffer, mpsc::error::SendError<Frame>>
+where
+    I: Iterator<Item = Frame>,
+{


Suggested change

/// Reports encode metrics for an already-encoded message and forwards all of

/// its frames to the websocket send task.

fn ws_forward_frames<I>(

metrics: &SendMetrics,

outgoing_frames: &mpsc::UnboundedSender<Frame>,

workload: Option<WorkloadType>,

num_rows: Option<usize>,

encoded: (EncodeMetrics, InUseSerializeBuffer, I),

) -> Result<InUseSerializeBuffer, mpsc::error::SendError<Frame>>

where

I: Iterator<Item = Frame>,

{

/// Reports encode metrics for an already-encoded message

/// and forwards all of its frames to the websocket send task.

fn ws_forward_frames(

metrics: &SendMetrics,

outgoing_frames: &mpsc::UnboundedSender<Frame>,

workload: Option<WorkloadType>,

num_rows: Option<usize>,

encoded: (EncodeMetrics, InUseSerializeBuffer, impl Iterator<Item = Frame>),

) -> Result<InUseSerializeBuffer, mpsc::error::SendError<Frame>> {

(not so important, just a nit)

Centril · 2026-04-10T06:48:12Z

crates/core/src/client/messages.rs

    }
 }

+fn finalize_binary_serialize_buffer(


Suggested change

fn finalize_binary_serialize_buffer(

/// Finalizes a binary-encoded message from the server

/// stored in `buffer` by potentially compressing the `buffer`

/// depending on `uncompressed_len` and `compression`.

fn finalize_binary_serialize_buffer(

Centril · 2026-04-10T07:16:59Z

crates/core/src/client/messages.rs

+    // At this point, we no longer have a use for `msg`,
+    // so try to reclaim its buffers.
+    msg.consume_each_list(&mut |buffer| bsatn_rlb_pool.try_put(buffer));
+
+    let frame = ws_v3::ServerFrame::Single(inner.freeze());
+    let srv_msg = buffer.write_with_tag(ws_common::SERVER_MSG_COMPRESSION_TAG_NONE, |w| {
+        bsatn::to_writer(w.into_inner(), &frame).expect("should be able to bsatn encode v3 server frame");
+    });
+    let srv_msg_len = srv_msg.len();


Suggested change

// At this point, we no longer have a use for `msg`,

// so try to reclaim its buffers.

msg.consume_each_list(&mut |buffer| bsatn_rlb_pool.try_put(buffer));

let frame = ws_v3::ServerFrame::Single(inner.freeze());

let srv_msg = buffer.write_with_tag(ws_common::SERVER_MSG_COMPRESSION_TAG_NONE, |w| {

bsatn::to_writer(w.into_inner(), &frame).expect("should be able to bsatn encode v3 server frame");

});

let srv_msg_len = srv_msg.len();

let frame = ws_v3::ServerFrame::Single(inner.freeze());

let srv_msg = buffer.write_with_tag(ws_common::SERVER_MSG_COMPRESSION_TAG_NONE, |w| {

bsatn::to_writer(w.into_inner(), &frame).expect("should be able to bsatn encode v3 server frame");

});

let srv_msg_len = srv_msg.len();

// At this point, we no longer have a use for `msg` and `frame`,

// so try to reclaim their buffers.

msg.consume_each_list(&mut |buffer| bsatn_rlb_pool.try_put(buffer));

frame.consume_each_list(&mut |buffer| bsatn_rlb_pool.try_put(buffer));

We should reuse frame's buffers as well.

Centril · 2026-04-10T07:19:53Z

crates/core/src/client/messages.rs

+    msg: ws_v2::ServerMessage,
+    compression: ws_v1::Compression,
+) -> (InUseSerializeBuffer, Bytes) {
+    let mut inner = BytesMut::with_capacity(SERIALIZE_BUFFER_INIT_CAP);


Let's extract out from BsatnRowListBuilderPool::take_row_list_builder a method fn take_buffer(&self) -> BytesMut which you can then use here.

Suggested change

let mut inner = BytesMut::with_capacity(SERIALIZE_BUFFER_INIT_CAP);

let mut inner = bsatn_rlb_pool.take_buffer();

Centril · 2026-04-10T07:20:53Z

crates/client-api-messages/src/websocket/v3.rs

+pub enum ServerFrame {
+    /// A single logical server message.
+    Single(Bytes),
+    /// Multiple logical server messages that should be processed in-order.


Suggested change

/// Multiple logical server messages that should be processed in-order.

/// Multiple logical server messages that should be processed in-order.

///

/// This is currently never produced.

Centril · 2026-04-10T07:40:46Z

crates/client-api-messages/src/websocket/v3.rs

+/// The inner bytes are BSATN-encoded v2 [`crate::websocket::v2::ClientMessage`] values.
+#[derive(SpacetimeType, Debug)]
+#[sats(crate = spacetimedb_lib)]
+pub enum ClientFrame {


Hmm; both Single and Batch could actually be stored as just Bytes with the logic that the host will try to deserialize as many ClientMessages as it can until the read buffer has been exhausted.

Alternatively, we could go for a more structured / typed representation here, with Box<[super::v2::ClientMessage]>. This would still include some of the overhead/bookkeeping bytes and allocations which Mazdak's suggestion eliminates, but it would add type safety and eliminate a lot of ser/de boilerplate code. That would look like:

enum ClientFrame { Single(super::v2::ClientMessage), Batch(Box<[super::v2::ClientMessage]>), }

And, you know, doing the same thing to ServerMessage.

Yeah if we go for the tagged representation then a typed one seems better. However, given that the goal of this PR is perf, it seems to me that we should go for the representation with the least overhead. (This PR actually is a regression for the single case, but with my suggestion, the added overhead is a single branch rather than a full extra allocation / taking from the pool + memcpy.)

Centril · 2026-04-10T07:41:58Z

crates/client-api-messages/src/websocket/v3.rs

+/// The inner bytes are BSATN-encoded v2 [`crate::websocket::v2::ServerMessage`] values.
+#[derive(SpacetimeType, Debug)]
+#[sats(crate = spacetimedb_lib)]
+pub enum ServerFrame {


ServerFrame could also just be Bytes.

Centril · 2026-04-10T07:59:46Z

crates/core/src/client/message_handlers_v3.rs

+    let frame = match message {
+        DataMessage::Binary(message_buf) => bsatn::from_slice::<ws_v3::ClientFrame>(&message_buf)?,
+        DataMessage::Text(_) => {
+            return Err(MessageHandleError::TextDecode(serde_json::Error::custom(
+                "v3 websocket does not support text messages",
+            )))
+        }
+    };
+
+    match frame {
+        ws_v3::ClientFrame::Single(message) => {
+            let message = bsatn::from_slice::<ws_v2::ClientMessage>(&message)?;
+            super::message_handlers_v2::handle_decoded_message(client, message, timer).await?;
+        }
+        ws_v3::ClientFrame::Batch(messages) => {
+            for message in messages {
+                let message = bsatn::from_slice::<ws_v2::ClientMessage>(&message)?;
+                super::message_handlers_v2::handle_decoded_message(client, message, timer).await?;
+            }
+        }
+    }


If ClientFrame is just message: Bytes (directly from DataMessage), the V2 and V3 code for handle could be the same and we wouldn't need the temporary allocation.

Centril · 2026-04-10T08:22:37Z

crates/core/src/client/messages.rs

+///
+/// This mirrors the v2 framing by prepending the compression tag and applying
+/// conditional compression when configured.
+pub fn serialize_v3(


The comments I made here re. take_buffer and consume_each_list become moot though if ServerFrame becomes just Bytes. Then serialize_v3 == serialize_v2 and the only difference becomes on the client sides. We'll still need the v3 protocol so that newer clients know that they should deserialize multiple ServerMessages.

gefjon · 2026-04-10T15:29:21Z

crates/client-api-messages/src/websocket/v3.rs

+/// The inner bytes are BSATN-encoded v2 [`crate::websocket::v2::ClientMessage`] values.
+#[derive(SpacetimeType, Debug)]
+#[sats(crate = spacetimedb_lib)]
+pub enum ClientFrame {


Alternatively, we could go for a more structured / typed representation here, with Box<[super::v2::ClientMessage]>. This would still include some of the overhead/bookkeeping bytes and allocations which Mazdak's suggestion eliminates, but it would add type safety and eliminate a lot of ser/de boilerplate code. That would look like:

enum ClientFrame { Single(super::v2::ClientMessage), Batch(Box<[super::v2::ClientMessage]>), }

And, you know, doing the same thing to ServerMessage.

gefjon · 2026-04-10T15:30:31Z

crates/client-api-messages/src/websocket.rs

 pub mod common;
 pub mod v1;
 pub mod v2;
+pub mod v3;


Why do you prefer adding a new version with a wrapper, rather than adding new message variants to v2?

gefjon · 2026-04-10T15:31:59Z

crates/client-api-messages/src/websocket/v3.rs

+pub enum ServerFrame {
+    /// A single logical server message.
+    Single(Bytes),
+    /// Multiple logical server messages that should be processed in-order.


What are the intended semantics for ServerFrame::Batch? Will the server always respond ServerFrame::Single to ClientFrame::Single and ServerFrame::Batch to ClientFrame::Batch, or is the server free to re-group messages so long as order is preserved?

I was also wondering this xD

v3 websocket protocol

d494169

joshua-spacetime requested review from Centril and gefjon as code owners April 8, 2026 21:45

This was referenced Apr 8, 2026

Update typescript sdk to use v3 websocket api #4762

Open

Update rust sdk to use v3 websocket api #4765

Open

Update C# sdk to use v3 websocket api #4763

Open

Update unreal sdk to use v3 websocket api #4764

Open

joshua-spacetime requested a review from coolreader18 April 9, 2026 04:51

Centril requested changes Apr 10, 2026

View reviewed changes

gefjon reviewed Apr 10, 2026

View reviewed changes

-fn finalize_binary_serialize_buffer(
+/// Finalizes a binary-encoded message from the server
+/// stored in `buffer` by potentially compressing the `buffer`
+/// depending on `uncompressed_len` and `compression`.
+fn finalize_binary_serialize_buffer(

	let mut inner = BytesMut::with_capacity(SERIALIZE_BUFFER_INIT_CAP);
	let mut inner = bsatn_rlb_pool.take_buffer();

Conversation

joshua-spacetime commented Apr 8, 2026

Description of Changes

API and ABI breaking changes

Expected complexity level and risk

Testing

Uh oh!

Centril left a comment

Choose a reason for hiding this comment

Uh oh!

Centril Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Centril Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Centril Apr 10, 2026 •

edited

Loading

Centril Apr 10, 2026 •

edited

Loading