Zero-copy decoding

[SeanTAllen]

The problem

Every decode of a variable-length value (str, bin, ext) copies the data out of the reader's buffer. The stdlib Reader.block(len) allocates a new Array[U8] of size len and copies bytes into it from internal chunks — even when the requested range falls entirely within a single chunk. For the msgpack decoder, this means every decoded string, binary blob, and extension value triggers an allocation and memcpy proportional to its size.

For small values (field names, short strings), the cost is negligible. For workloads that decode large payloads — multi-KB binary data, long strings, bulk ext values — the copies add up. Zero-copy decoding eliminates this overhead by returning views into the reader's existing buffer instead of copies.

How other libraries handle this

MessagePack-CSharp

TryReadStringSpan returns a ReadOnlySpan<byte> that references the input buffer directly. Only works when the data falls within a contiguous segment of the input; returns false otherwise, and the caller falls back to a copying read.

msgpack-ruby

Uses a size threshold. Below read_reference_threshold bytes, the decoder copies (because small copies are cheaper than reference management overhead). Above the threshold, it returns a reference into the input buffer. The threshold is configurable.

rmp-serde (Rust)

Zero-copy via the ReadSlice trait. The deserializer can borrow &[u8] from the input slice. The deserialized values have lifetime bounds tied to the input — they can't outlive the buffer they reference. Requires the input to be a contiguous slice.

Common pattern

All implementations share the same constraint: zero-copy only works when the requested data is contiguous in the input buffer. When data spans buffer boundaries, a copy is unavoidable.

Why this is natural in Pony

Pony's stdlib provides two building blocks that make zero-copy straightforward:

Array[U8] val.trim(from, to) returns a shared portion of the array. Both the original and the new array are immutable (val), as they share memory. The operation does not allocate a new array pointer nor copy elements. From the stdlib documentation:

Return a shared portion of this array, covering from until to. Both the original and the new array are immutable, as they share memory. The operation does not allocate a new array pointer nor copy elements.

String.from_array(data: Array[U8] val) creates a String val that reuses the underlying data pointer of the input array. No copy.

Chained together, these give a complete zero-copy path from input buffer to decoded value:

input chunk (Array[U8] val)
  -> trim(offset, offset + len)  -- zero-copy view
  -> String.from_array(view)     -- zero-copy String

No data copies at any step. Both trim and from_array allocate small header objects (array/string metadata), but the element data is shared — no memcpy of the payload bytes.

Why the stdlib Reader can't do this

The stdlib Reader.block(len) returns Array[U8] iso^. The iso capability means isolated — the caller has exclusive access, and no other reference to that memory can exist. This is useful when the caller needs to mutate the result, but it means block() must copy data into a freshly allocated array rather than sharing the reader's internal buffer.

The reader internally stores chunks as Array[U8] val (immutable, shareable). The data is already in the right capability for sharing — block() copies it out into iso unnecessarily for callers who only need val.

A zero-copy reader that returns Array[U8] val instead of Array[U8] iso^ can use trim directly on its internal chunks, avoiding the copy entirely.

Design

Standalone reader

A new ZeroCopyReader class in the msgpack package. Not a replacement for buffered.Reader — a separate type with its own API. The existing MessagePackDecoder and MessagePackStreamingDecoder continue using the stdlib Reader unchanged.

class ZeroCopyReader
  """
  A reader optimized for zero-copy extraction of byte ranges.
  Returns `val` views into the internal buffer instead of
  copying into fresh `iso` arrays.

  Zero-copy succeeds when the requested range falls within a
  single internal chunk. When data spans chunk boundaries, the
  reader falls back to copying.
  """
  embed _chunks: Array[(Array[U8] val, USize)]
  var _available: USize = 0

  fun ref append(data: Array[U8] val) =>
    """
    Append a chunk. The reader holds a `val` reference to the
    chunk — the caller retains theirs. No copy.
    """
    _available = _available + data.size()
    _chunks.push((data, 0))

  fun ref block(len: USize): Array[U8] val ? =>
    """
    Return `len` bytes. If the range falls within a single
    chunk, returns a `trim`'d view (zero-copy). If it spans
    chunks, copies into a new array (fallback).
    """
    if _available < len then error end
    _available = _available - len

    (let data, let offset) = _chunks(0)?
    let avail = data.size() - offset
    if avail >= len then
      // Fast path: single chunk, zero-copy
      let result = data.trim(offset, offset + len)
      if avail == len then
        _chunks.shift()?
      else
        _chunks(0)? = (data, offset + len)
      end
      result
    else
      // Slow path: spans chunks, must copy
      _block_copy(len)?
    end

  fun ref _block_copy(len: USize): Array[U8] val ? =>
    // ... copy from multiple chunks, same as stdlib Reader ...

The reader uses chunk-based storage similar to the stdlib Reader but returns val instead of iso^ from block(). The stdlib uses a doubly-linked List for O(1) head removal; the sketch above uses Array for simplicity but an implementation should consider List for the same reason. It also needs the same supporting methods the msgpack decoder uses: u8(), u16_be(), u32_be(), u64_be(), i8(), i16_be(), i32_be(), i64_be(), f32_be(), f64_be(), peek_u8(offset), peek_u16_be(offset), peek_u32_be(offset), size(), and skip(n).

For the fixed-size numeric reads (u8, u16_be, etc.), zero-copy doesn't matter — they read 1-8 bytes. These can copy.

Decoder integration

The existing MessagePackDecoder takes buffered.Reader (a concrete class). Without interface abstraction, a new decoder primitive is needed for the zero-copy reader:

primitive MessagePackZeroCopyDecoder
  fun str(b: ZeroCopyReader): String val ? =>
    let t = _read_type(b)?
    let len = if (t and 0xE0) == _FormatName.fixstr() then
      (t and 0x1F).usize()
    elseif t == _FormatName.str_8() then
      b.u8()?.usize()
    elseif t == _FormatName.str_16() then
      b.u16_be()?.usize()
    elseif t == _FormatName.str_32() then
      b.u32_be()?.usize()
    else
      error
    end
    String.from_array(b.block(len)?)

The decode logic is identical to MessagePackDecoder — only the reader type and return capability differ. The key change is String.from_array(b.block(len)?) instead of String.from_iso_array(b.block(len)?).

This duplication is the cost of not abstracting over reader types. The alternative — a common interface — is deferred per the substitutability decision.

Which formats benefit

Only variable-length formats benefit from zero-copy:

Format family	Current return	Zero-copy return	Benefit
str (fixstr, str_8/16/32)	String iso^	String val	Avoids copy of string data
bin (bin_8/16/32)	Array[U8] iso^	Array[U8] val	Avoids copy of binary data
ext (fixext, ext_8/16/32)	(U8, Array[U8] val)	Same	Avoids copy of ext data

Fixed-size formats (nil, bool, integers, floats) read at most 9 bytes. Zero-copy overhead (trim bookkeeping) would exceed the copy cost at these sizes. These methods can copy unconditionally.

Single-chunk vs. multi-chunk

The zero-copy fast path only works when the requested range falls within a single internal chunk. When it spans chunks, the reader must copy. How often the fast path hits depends on how data arrives:

• Large chunks (e.g., full TCP reads of 4-64 KB): Most values fall within a single chunk. Zero-copy hit rate is high.
• Small fragments (e.g., byte-at-a-time): Nearly every multi-byte value spans chunks. Zero-copy hit rate is near zero.

Real-world network I/O typically delivers data in chunks of hundreds to thousands of bytes, so the fast path should hit for the majority of values in typical use.

Chunk consolidation option

An alternative to accepting multi-chunk fallbacks: consolidate on append. When the caller appends data, copy it into a single growing buffer. This is one copy on input, but then all reads are guaranteed zero-copy (single chunk).

fun ref append(data: Array[U8] val) =>
  // Copy into a single contiguous buffer
  _buffer.append(data)

fun ref block(len: USize): Array[U8] val ? =>
  // Always single-chunk, always zero-copy
  let result = _buffer.trim(_offset, _offset + len)
  _offset = _offset + len
  result

The tradeoff: one copy on input (into the consolidated buffer) vs. potentially zero copies on read. Whether this wins depends on how many values are decoded from each appended chunk. If many values are decoded per chunk, consolidation amortizes the input copy across many zero-copy reads. If few values are decoded per chunk, it may be worse than the current approach.

There's a subtlety: _buffer must be val for trim to work. But we're appending to it, which requires mutability. The pattern would be: accumulate into a ref or iso array, then freeze it to val before serving reads. This means consolidation happens in batches — append data, freeze, serve reads, append more, freeze again. Each freeze produces a new val array that covers the latest data.

Streaming decoder variant

A zero-copy streaming decoder would pair the ZeroCopyReader with the streaming decode logic. The existing MessagePackStreamingDecoder uses buffered.Reader internally and returns MessagePackValue (which includes String val and Array[U8] val). A zero-copy variant would use ZeroCopyReader instead and return the same types — but the String val and Array[U8] val values would reference the reader's buffer instead of being independent copies.

The caller-facing API would be identical. The only observable difference is performance and the lifetime relationship between decoded values and the reader's buffer.

Lifetime considerations

Zero-copy means decoded values hold references to the reader's internal chunks. As long as any decoded value is live, the chunk it references stays in memory (Pony's GC handles this — val references are traced normally).

In the normal case, this is fine — decoded values are processed and discarded, and the chunks become eligible for GC. But if the caller holds decoded values for a long time (e.g., caching them), the entire original chunk stays alive even if only a few bytes from it are referenced. This is the classic "small slice holds large buffer alive" problem.

For most msgpack use cases (decode, process, discard), this isn't a concern. For callers who cache decoded values, the tradeoff is explicit: use zero-copy for throughput, accept that cached values pin their source chunks.

Scope

This proposal covers:

• A ZeroCopyReader class in the msgpack package
• A MessagePackZeroCopyDecoder primitive (or equivalent) for decoding from it
• The zero-copy path via Array.trim and String.from_array
• Discussion of chunk consolidation as an optimization

This proposal does not cover:

• Substitutability with buffered.Reader (deferred)
• Changes to existing MessagePackDecoder or MessagePackStreamingDecoder
• A common reader interface

[SeanTAllen]

documentation would be very important to this. to let folks know a zero copy version exists, but you have to use our special buffer class rather than Reader and that could impact interop with other classes.

[SeanTAllen]

i don't see the need to a special zero copy streaming one. that really is an implementation detail. we can just make the streaming one always use the zero copy buffering.

[SeanTAllen]

Re: Standalone reader section, which notes:

The stdlib uses a doubly-linked List for O(1) head removal; the sketch above uses Array for simplicity but an implementation should consider List for the same reason.

Array is the right choice here. The O(1) shift advantage of List doesn't matter in practice because the chunk count is small — network I/O delivers data in reasonably sized chunks, and they get consumed roughly as fast as they arrive.

Looking at the full access pattern from the sketch:

• Read front (_chunks(0)?) — every block() call. O(1) for Array.
• Update front in place (_chunks(0)? = (data, offset + len)) — most block() calls. O(1) for Array.
• Sequential scan from front — _block_copy walks multiple chunks, and peek_* methods find bytes at arbitrary offsets by scanning chunks. Array's cache locality wins here; List chases pointers.
• Push back — on each append(). Amortized O(1) for both.
• Shift front — only when a chunk is fully consumed. O(n) for Array, but n is small (typically a handful of chunks), so the cost is negligible.

The dominant operations (front read, front update, sequential scan) all favor Array or are neutral. Shift is the only one that favors List, and with small n it's not meaningful.

[SeanTAllen]

Re: Chunk consolidation option section.

Going with multi-chunk with fallback. No copy on input, zero-copy when the read falls within a single chunk (common case with typical network I/O), copy fallback when spanning chunks — which is no worse than what buffered.Reader does today.

Consolidation is a potential future optimization. Opened #74 to track it.

[SeanTAllen]

This is addressed via #75