Visitor/SAX-style parse API

[SeanTAllen]

The problem

The streaming decoder returns a DecodeResult union for each
value. The caller uses match to dispatch on the decoded type.
This works well when every value in the stream matters, but it
has two costs:

1. Intermediate objects for every value. Each decoded value
   becomes a Pony object -- a String val, an Array[U8] val,
   a MessagePackTimestamp, etc. For callers that want to
   extract a few fields from a large message, or transform
   msgpack directly to another format, most of these objects are
   created only to be immediately discarded.

2. Explicit container management. When the decoder returns a
   MessagePackArray or MessagePackMap header, the caller is
   responsible for tracking how many elements remain to be read
   and managing the nesting. For deeply nested structures, this
   means manually maintaining a stack of element counts.

A visitor API inverts the control flow. Instead of the caller
pulling values and dispatching on type, the parser drives the
traversal and calls methods on a visitor object as it encounters
each element. The visitor implements only the callbacks it cares
about; unneeded values are never materialized into Pony objects.

Use cases

Selective field extraction. A message contains a large map
with dozens of keys. The caller needs only two of them. With the
current API, the caller must decode every key and value
(or use skip to advance past unwanted values) to traverse
the map. With a visitor, the parser handles the traversal and
calls visit_str for each key -- the visitor checks if it's a
key of interest and sets a flag to capture the next value.

Format transcoding. Converting msgpack directly to JSON,
CBOR, or another wire format. The visitor's callbacks map
directly to the output format's write operations
(visit_nil writes null, start_array writes [,
visit_str writes a quoted string, etc.) with no intermediate
representation.

Validation and metrics. Scanning a message to check
structure (correct nesting, expected types in expected
positions) or to collect statistics (count of each type,
maximum nesting depth, total string bytes) without building
any data structures.

Streaming transformation. Processing a stream of msgpack
messages where each message is transformed or filtered and
forwarded. The visitor can write directly to an output buffer
during traversal.

How other libraries handle this

msgpack-cxx (C++)

The C++ library provides a full visitor interface driven by
msgpack::parse(). The visitor is a class satisfying a concept
(duck-typed template) with methods for each msgpack type. A
msgpack::null_visitor base class provides default
implementations that return true (continue parsing) for all
methods.

Leaf value callbacks:

bool visit_nil();
bool visit_boolean(bool v);
bool visit_positive_integer(uint64_t v);
bool visit_negative_integer(int64_t v);
bool visit_float32(float v);
bool visit_float64(double v);
bool visit_str(const char* v, uint32_t size);
bool visit_bin(const char* v, uint32_t size);
bool visit_ext(const char* v, uint32_t size);

Container callbacks have start/end pairs with item-level
hooks for maps:

bool start_array(uint32_t num_elements);
bool start_array_item();
bool end_array_item();
bool end_array();

bool start_map(uint32_t num_kv_pairs);
bool start_map_key();
bool end_map_key();
bool start_map_value();
bool end_map_value();
bool end_map();

Error callbacks:

void parse_error(size_t parsed_offset, size_t error_offset);
void insufficient_bytes(size_t parsed_offset, size_t error_offset);

Every visit_*/start_*/end_* method returns bool. The
parse function checks this return value after each callback --
returning false from any callback aborts parsing immediately.
The parse function itself returns true if parsing completed
successfully, false otherwise.

Key design points: the parser owns the read loop and drives
all recursion into containers. The visitor never reads from the
buffer directly. Container callbacks receive the element count
up front (start_array(num_elements)) so the visitor can
pre-allocate if desired.

Serde (Rust)

Serde's Deserializer/Visitor split is the most influential
visitor-based deserialization design in modern use. The
Visitor trait has methods for each data type, all with
default implementations that return a type error:

fn visit_bool(self, v: bool) -> Result<Self::Value, E>;
fn visit_i64(self, v: i64) -> Result<Self::Value, E>;
fn visit_u64(self, v: u64) -> Result<Self::Value, E>;
fn visit_f64(self, v: f64) -> Result<Self::Value, E>;
fn visit_str(self, v: &str) -> Result<Self::Value, E>;
fn visit_bytes(self, v: &[u8]) -> Result<Self::Value, E>;
fn visit_seq(self, seq: A) -> Result<Self::Value, E>;
fn visit_map(self, map: A) -> Result<Self::Value, E>;
fn visit_none(self) -> Result<Self::Value, E>;
// ... more

Default implementations forward smaller integer types to
larger ones (e.g., visit_i8 calls visit_i64), so
implementors only need to handle the widest types. The
expecting() method provides error messages.

Key difference from msgpack-cxx: Serde's visitor produces a
value (Self::Value), making it a construction-oriented
pattern. The callbacks for sequences and maps receive iterator
objects (SeqAccess, MapAccess) that the visitor pulls from,
rather than receiving push-style start/end notifications. This
pull-within-push hybrid gives the visitor control over
iteration within each container.

Jackson (Java)

Jackson's JsonParser is a pull-based streaming API rather
than a visitor, but it solves the same problem -- processing
serialized data without building a tree. The caller advances
a cursor with nextToken() and dispatches on the token type:

while (parser.nextToken() != JsonToken.END_OBJECT) {
    String field = parser.currentName();
    parser.nextToken();
    if ("name".equals(field)) {
        name = parser.getText();
    } else {
        parser.skipChildren();
    }
}

skipChildren() is the skip equivalent for containers --
it advances past the current array or object and all its
contents without materializing them.

Jackson's approach is simpler to implement (no visitor trait,
just a token stream) but puts the traversal burden on the
caller. Every consumer must handle the token loop, nesting,
and error recovery. The visitor pattern inverts this: the
parser handles traversal and the consumer handles only the
values it cares about.

Common patterns across libraries

All of these designs share a few properties:

1. The parser drives the read loop. The consumer does not
   interact with the underlying byte buffer.
2. Early termination. The consumer can signal "stop" (via
   return value or exception) to abort parsing without reading
   the rest of the data.
3. Default no-ops or errors. Callbacks the consumer doesn't
   implement either succeed silently (msgpack-cxx) or fail with
   a type error (Serde). The choice depends on whether the
   pattern is "process what you understand" (no-ops) or
   "construct a specific type" (errors).
4. Container boundaries are explicit. Start/end pairs for
   arrays and maps, with the element count available at
   start_* time.

Design

Visitor trait

A Pony trait with default no-op implementations. Callers
implement only the callbacks they need. All methods use a
ref receiver so the visitor can accumulate state.

Each callback returns Bool -- true to continue parsing,
false to abort. This matches msgpack-cxx's approach and is
the right default for Pony: the "process what you understand,
skip what you don't" model suits a wire format library where
forward compatibility matters.

trait ref MessagePackVisitor
  """
  Callback interface for visitor-style traversal of msgpack
  data. Implement only the methods you need; unimplemented
  methods return `true` (continue parsing).

  Every method returns `Bool`. Return `true` to continue
  parsing or `false` to abort. When any method returns
  `false`, the parse function returns `false` immediately.
  """
  // Leaf values
  fun ref visit_nil(): Bool => true
  fun ref visit_bool(v: Bool): Bool => true
  fun ref visit_uint(v: U64): Bool => true
  fun ref visit_int(v: I64): Bool => true
  fun ref visit_float_32(v: F32): Bool => true
  fun ref visit_float_64(v: F64): Bool => true
  fun ref visit_str(v: String val): Bool => true
  fun ref visit_bin(v: Array[U8] val): Bool => true
  fun ref visit_ext(
    ext_type: U8,
    data: Array[U8] val)
    : Bool
  =>
    true
  fun ref visit_timestamp(sec: I64, nsec: U32): Bool =>
    true

  // Array container
  fun ref start_array(size: U32): Bool => true
  fun ref start_array_item(): Bool => true
  fun ref end_array_item(): Bool => true
  fun ref end_array(): Bool => true

  // Map container
  fun ref start_map(size: U32): Bool => true
  fun ref start_map_key(): Bool => true
  fun ref end_map_key(): Bool => true
  fun ref start_map_value(): Bool => true
  fun ref end_map_value(): Bool => true
  fun ref end_map(): Bool => true

  // Error reporting
  fun ref parse_error(): None => None
  fun ref insufficient_data(): None => None

Design decisions in the trait

Unified integer callbacks (visit_uint/visit_int) instead
of per-width callbacks. The msgpack wire format encodes
integers in multiple widths (fixint, uint_8, uint_16, ...) but
these are transparent size optimizations. A visitor extracting a
"count" field doesn't care whether the sender used uint_8 or
uint_32. Using U64 and I64 collapses the format families
into the values they represent, matching how
MessagePackDecoder.uint and MessagePackDecoder.int already
work.

If a caller needs the original wire format, they should use the
lower-level decoder API directly. The visitor pattern is for
callers who care about values, not encodings.

Separate visit_float_32/visit_float_64 instead of a
unified float callback. Unlike integers where smaller formats
are strict subsets of larger ones, F32 and F64 have
different precision. Collapsing F32 into F64 would silently
change the precision semantics. Keeping them separate lets the
visitor handle each precision explicitly.

visit_str receives String val, not raw bytes and a
length. The C++ library passes (const char*, uint32_t) for
zero-copy access to the buffer. In Pony, MessagePackDecoder
produces String iso^ from the reader -- the data is copied
out of the reader's internal buffer into an iso array, then
String.from_iso_array takes ownership without a second copy.
Passing a String val to the visitor is idiomatic Pony.
For true zero-copy semantics,
MessagePackZeroCopyDecoder produces String val views
directly into the reader's buffer (when the data falls within
a single chunk; it falls back to copying when data spans
chunk boundaries). A visitor parser built on the zero-copy
decoder would avoid allocation for most string values
(see "Relationship to existing APIs" below).

Explicit visit_timestamp. Timestamps are semantically
distinct from extensions even though they are wire-encoded as
ext type -1. The streaming decoder already decodes them into
MessagePackTimestamp rather than MessagePackExt. The visitor
follows the same principle -- timestamps get their own callback
so the visitor doesn't need to check for ext type 0xFF
manually.

visit_ext does not receive timestamps. When the parser
encounters an ext value with type 0xFF (timestamp), it calls
visit_timestamp, not visit_ext. This is consistent with
the streaming decoder's behavior and the msgpack spec's
treatment of timestamps as a distinct type.

Map item callbacks (start_map_key/end_map_key/
start_map_value/end_map_value). These bracket each key
and value within a map traversal, following msgpack-cxx's
design. They let the visitor know whether the next value is
a key or a value without maintaining a toggle flag. The
sequence for each map entry is:

start_map_key -> visit_*(key) -> end_map_key ->
  start_map_value -> visit_*(value) -> end_map_value

Error callbacks return None, not Bool. Once an error
occurs, parsing cannot continue -- the stream is in an
indeterminate state. The callbacks are informational. The
parse function returns false after calling them regardless
of what the visitor does.

The parse function

A standalone function (or method on a new primitive) that
drives the traversal:

primitive MessagePackVisitorParser
  fun parse(
    reader: Reader ref,
    visitor: MessagePackVisitor ref)
    : Bool
  =>
    """
    Parse one complete msgpack value from the reader,
    calling visitor methods for each element encountered.
    Returns `true` if parsing completed successfully,
    `false` if the visitor aborted or an error occurred.

    For containers, this recursively parses all contained
    elements, calling start/end callbacks around each
    level.
    """
    // peek at format byte, dispatch to appropriate
    // _parse_* method
    try
      _parse_value(reader, visitor)?
    else
      false
    end

  fun _parse_value(
    reader: Reader ref,
    visitor: MessagePackVisitor ref)
    : Bool ?
  =>
    let fb = reader.peek_u8()?
    if fb <= 0x7F then
      // positive fixint
      visitor.visit_uint(reader.u8()?.u64())
    elseif fb <= 0x8F then
      // fixmap
      _parse_map(reader, visitor)?
    elseif fb <= 0x9F then
      // fixarray
      _parse_array(reader, visitor)?
    elseif fb <= 0xBF then
      // fixstr
      let s = MessagePackDecoder.fixstr(reader)?
      visitor.visit_str(consume s)
    elseif fb == _FormatName.nil() then
      MessagePackDecoder.nil(reader)?
      visitor.visit_nil()
    elseif (fb == _FormatName.falsey())
      or (fb == _FormatName.truthy())
    then
      visitor.visit_bool(
        MessagePackDecoder.bool(reader)?)
    // ... remaining format bytes dispatched similarly
    else
      error
    end

  fun _parse_array(
    reader: Reader ref,
    visitor: MessagePackVisitor ref)
    : Bool ?
  =>
    let count = MessagePackDecoder.array(reader)?
    if not visitor.start_array(count) then
      return false
    end
    var i: U32 = 0
    while i < count do
      if not visitor.start_array_item() then
        return false
      end
      if not _parse_value(reader, visitor)? then
        return false
      end
      if not visitor.end_array_item() then
        return false
      end
      i = i + 1
    end
    visitor.end_array()

  fun _parse_map(
    reader: Reader ref,
    visitor: MessagePackVisitor ref)
    : Bool ?
  =>
    let count = MessagePackDecoder.map(reader)?
    if not visitor.start_map(count) then
      return false
    end
    var i: U32 = 0
    while i < count do
      if not visitor.start_map_key() then
        return false
      end
      if not _parse_value(reader, visitor)? then
        return false
      end
      if not visitor.end_map_key() then
        return false
      end
      if not visitor.start_map_value() then
        return false
      end
      if not _parse_value(reader, visitor)? then
        return false
      end
      if not visitor.end_map_value() then
        return false
      end
      i = i + 1
    end
    visitor.end_map()

Who drives the read loop

The parse function owns the reader and drives all reads.
The visitor never sees the reader. This is a deliberate
constraint:

• The visitor can't accidentally corrupt the reader's state.
• The parser handles all format byte dispatching, length
  reading, and container recursion.
• The visitor is a pure callback object -- it receives
  decoded values and makes decisions (continue or abort).

This means the visitor cannot do things like "read the next
3 values manually" -- it must accept them as they arrive
through callbacks. If a caller needs that level of control,
the lower-level MessagePackDecoder or
MessagePackStreamingDecoder APIs are the right choice.

Error handling and early termination

Two mechanisms for stopping the parse:

Visitor returns false. The parse function checks the
return value after every callback. When any callback returns
false, the parse function returns false immediately
without reading further. This is the clean abort path --
the visitor has seen enough and wants to stop. The reader
is left positioned after the last fully-read value. For
non-streaming use, the caller can discard the reader. For
streaming use, the reader's position is indeterminate within
the current top-level value.

Read error (partial). If the reader doesn't have enough
data, MessagePackDecoder methods raise error, which
propagates through _parse_value. The parse function catches
this at the top level, calls visitor.insufficient_data(),
and returns false. The reader's state may be corrupted
(same limitation as MessagePackDecoder -- see issue #14).

Invalid format byte. If the format byte is 0xC1 (the
only invalid byte in msgpack), the parser calls
visitor.parse_error() and returns false.

Relationship to existing APIs

The visitor parser builds on MessagePackDecoder. It uses the
decoder's methods to read values from the reader and adds the
traversal logic (container recursion, visitor callbacks) on
top. It does not replace or modify any existing API.

A zero-copy variant could be built on
MessagePackZeroCopyDecoder and ZeroCopyReader instead.
The zero-copy decoder returns String val and
Array[U8] val views into the reader's buffer when the
requested range falls within a single internal chunk,
avoiding allocation entirely. When data spans chunk
boundaries, it falls back to copying. This would be a
natural fit for visitors that transform or forward data
without retaining it, since the decoded values can be
released as soon as the callback returns. The tradeoff is
that decoded values pin the reader's underlying chunks in
memory until they are collected -- the same tradeoff
documented in the zero-copy decoder's own API.

The visitor parser does not build on
MessagePackStreamingDecoder. The streaming decoder's
peek-before-consume contract adds overhead that the visitor
doesn't need -- if the caller has all the data available (the
same assumption MessagePackDecoder makes), the visitor can
read directly.

The existing skip method on MessagePackDecoder serves a
related but distinct purpose. skip advances past a value
without decoding it, which is useful when the caller knows
it doesn't need a value. The visitor parser handles
skipping differently: values the visitor doesn't care about
are still decoded (to advance the reader), but the visitor's
no-op default callbacks discard them without the caller
needing to write any code. For callers that want to avoid
even the decoding cost, a future optimization could have the
parse function call skip when it can determine the visitor
has no interest in a value, but this would require the
visitor to signal disinterest before the value is read
(a more complex protocol than simple Bool returns).

Streaming-safe visitor parsing

A streaming-safe variant could be built on
MessagePackStreamingDecoder in the future. The streaming
decoder now provides the building blocks this would need:
peek-before-consume semantics that return NotEnoughData
without corrupting the reader, configurable size and depth
limits via MessagePackDecodeLimits, a skip method with
its own max_skip_values limit, automatic container depth
tracking, and opt-in UTF-8 validation.

The main challenge remains the parse function's recursion
stack. A non-streaming visitor parser uses the Pony call
stack to track its position within nested containers
(_parse_array calls _parse_value which calls
_parse_map, etc.). When the streaming decoder returns
NotEnoughData, the parse function must save this entire
recursion state so it can resume after more data arrives.
This requires converting the implicit call stack into an
explicit state machine -- a container stack tracking
(container type, total count, elements processed) at each
nesting level.

The streaming decoder's existing depth tracking and limit
enforcement would carry over directly, but the visitor
parser would need its own resumable traversal logic on top.
A streaming-safe visitor would also need additional error
callbacks beyond parse_error and insufficient_data:
InvalidUtf8 (returned by the streaming decoder when
UTF-8 validation is enabled) and LimitExceeded (returned
when size or depth limits are exceeded, or when skip
exceeds max_skip_values). This is deferred as future
work.

Capability considerations

The visitor trait uses ref receiver on all methods. This is
the right choice for stateful visitors that accumulate results
(extracted fields, transcoded output, statistics). A box
receiver would prevent the visitor from storing state, which
defeats the purpose for most use cases.

The parse function takes MessagePackVisitor ref, not iso
or val. Since the parser and visitor operate within the same
actor's context (the parser reads from a Reader ref, which
is already non-sendable), ref is the natural capability.

Example: selective field extraction

Extracting name and age from a msgpack map that may
contain other fields:

use "buffered"
use "msgpack"

class FieldExtractor is MessagePackVisitor
  var name: (String val | None) = None
  var age: (U64 | None) = None
  var _capture_name: Bool = false
  var _capture_age: Bool = false

  fun ref visit_str(v: String val): Bool =>
    if _capture_name then
      name = v
      _capture_name = false
    else
      // Check if this string is a map key we want
      match v
      | "name" => _capture_name = true
      | "age" => _capture_age = true
      end
    end
    true

  fun ref visit_uint(v: U64): Bool =>
    if _capture_age then
      age = v
      _capture_age = false
    end
    true

  fun ref start_map_key(): Bool =>
    // Reset capture flags at each new key
    _capture_name = false
    _capture_age = false
    true

  fun ref start_map_value(): Bool =>
    // Flags were set by visit_str during key phase
    true

Usage:

let reader: Reader ref = Reader
// ... append msgpack-encoded map data to reader ...
let extractor = FieldExtractor
let ok = MessagePackVisitorParser.parse(reader, extractor)
// extractor.name and extractor.age now hold the values
// (or None if the fields were not present)

Example: msgpack-to-JSON transcoding

Converting a msgpack value to a JSON string:

use "buffered"
use "msgpack"

class JsonTranscoder is MessagePackVisitor
  let _out: Writer ref
  var _first_item: Bool = true

  new create(out: Writer ref) =>
    _out = out

  fun ref visit_nil(): Bool =>
    _out.write("null")
    true

  fun ref visit_bool(v: Bool): Bool =>
    _out.write(if v then "true" else "false" end)
    true

  fun ref visit_uint(v: U64): Bool =>
    _out.write(v.string())
    true

  fun ref visit_int(v: I64): Bool =>
    _out.write(v.string())
    true

  fun ref visit_str(v: String val): Bool =>
    _out.write("\"")
    _out.write(v)
    _out.write("\"")
    true

  fun ref start_array(size: U32): Bool =>
    _out.write("[")
    _first_item = true
    true

  fun ref start_array_item(): Bool =>
    if not _first_item then _out.write(",") end
    _first_item = false
    true

  fun ref end_array(): Bool =>
    _out.write("]")
    true

  fun ref start_map(size: U32): Bool =>
    _out.write("{")
    _first_item = true
    true

  fun ref start_map_key(): Bool =>
    if not _first_item then _out.write(",") end
    _first_item = false
    true

  fun ref end_map_key(): Bool =>
    _out.write(":")
    true

  fun ref end_map(): Bool =>
    _out.write("}")
    true

Note: the JSON transcoder example omits string escaping
for brevity. A production implementation would need to
escape control characters, quotes, and backslashes in
visit_str.

Scope

This proposal covers:

• A MessagePackVisitor trait with default no-op methods
  for all msgpack value types and container boundaries.
• A MessagePackVisitorParser primitive with a parse
  method that drives the traversal.
• Non-streaming operation only (assumes all data is
  available, same as MessagePackDecoder).

This proposal does not cover:

• A streaming-safe visitor parser (resumable parsing on
  NotEnoughData). This requires a state machine to save
  and restore the recursion stack and is deferred as
  future work.
• Visitor-driven encoding (a MessagePackVisitorEncoder
  that accepts visitor-style calls and writes msgpack).
  This is the inverse pattern and may be useful for format
  transcoding but is a separate design.
• A zero-copy variant built on MessagePackZeroCopyDecoder
  and ZeroCopyReader. The design would be structurally
  identical to the MessagePackDecoder-based parser, with
  different reader and decoder types. Worth considering if
  the visitor parser proves useful and allocation
  overhead matters in practice.
• Integration with MessagePackDecoder.skip. The visitor's
  no-op default callbacks already handle unwanted values
  without caller effort. A future optimization could use
  skip to avoid decoding values the visitor provably
  ignores, but this would require a richer signaling
  protocol than the current Bool returns (see
  "Relationship to existing APIs").