UTF-8 validation for str format values

[SeanTAllen]

The problem

The MessagePack spec defines the str format family as storing
"a byte array as a UTF-8 string." The library currently treats
str values as opaque byte sequences -- the encoder accepts any
ByteSeq in its str methods, the decoders return a String
constructed directly from the raw bytes via
String.from_iso_array or String.from_array, and neither
side checks whether the bytes are valid UTF-8.

This is a spec compliance gap. A caller reading the spec would
expect str values to contain valid UTF-8, and the library does
nothing to enforce or verify that. The spec does acknowledge
the issue:

String objects may contain invalid byte sequence and the
behavior of a deserializer depends on the actual
implementation when it received invalid byte sequence.
Deserializers should provide functionality to get the
original byte array so that applications can decide how to
handle the object.

So the spec explicitly leaves validation policy to the
implementation, but recommends that deserializers provide access
to the raw bytes. This means the library needs a deliberate
policy rather than the current accidental permissiveness.

The gap affects both sides:

• Decoders: The library has two decoder primitives.
  MessagePackDecoder operates on buffered.Reader and returns
  String iso^ via String.from_iso_array.
  MessagePackZeroCopyDecoder operates on ZeroCopyReader and
  returns String val via String.from_array (zero-copy path).
  Neither performs UTF-8 validation. The streaming decoder
  (MessagePackStreamingDecoder) delegates to
  MessagePackZeroCopyDecoder internally.
• Encoder: MessagePackEncoder.str() and all format-specific
  str methods accept any ByteSeq. A caller can encode arbitrary
  binary data using the str format, producing messages that
  violate the spec's intent. The spec provides the bin format
  family specifically for raw binary data.

Pony's String type is a raw byte array with no UTF-8
guarantee. String.from_iso_array and String.from_array reuse
the underlying data pointer without any validation. The utf32
method can decode individual codepoints and returns 0xFFFD (the
Unicode replacement character) with a length of 1 for invalid
byte sequences, but there is no dedicated is_valid_utf8 method
in the stdlib. Validation would require walking the string byte
by byte using utf32 and checking for replacement characters, or
implementing a standalone UTF-8 validator.

How other libraries handle this

Python msgpack

The raw parameter controls whether str format values are
decoded as Python bytes (raw bytes) or Python str (Unicode).
The default is raw=False, which decodes str values as Python
str by applying UTF-8 decoding. If the bytes are not valid
UTF-8, a UnicodeDecodeError is raised unless the caller sets
unicode_errors to a non-strict handler (e.g., 'replace' or
'ignore').

The strict_map_key parameter (default True) restricts map
keys to specific types. For encoding, use_bin_type=True
(default since 1.0) encodes Python str as msgpack str and
Python bytes as msgpack bin.

The result: Python msgpack validates UTF-8 by default on decode,
with an escape hatch for invalid data. On encode, the type
system naturally separates str from bytes.

JavaScript @msgpack/msgpack

By default, str format values are decoded as JavaScript strings
using TextDecoder, which performs UTF-8 decoding. The
rawStrings: true decode option skips UTF-8 decoding and returns
Uint8Array instead of string for str values.

JavaScript's TextDecoder replaces invalid UTF-8 sequences with
the replacement character by default, or can be configured to
throw on invalid input via the fatal option.

Rust rmp / rmp-serde

The rmp crate provides low-level read_str and read_str_ref
functions that decode str format values as Rust &str, which
requires valid UTF-8. Attempting to decode invalid UTF-8 returns
a DecodeStringError::InvalidUtf8 error.

For cases where the data might contain invalid UTF-8, the crate
provides Raw and RawRef wrapper types that can hold either a
valid String or raw bytes with an associated UTF-8 error.
Callers choose which type to deserialize into based on whether
they expect valid UTF-8.

The result: Rust enforces UTF-8 validity by default because the
language's str type requires it. The Raw/RawRef types are
the escape hatch for working with potentially invalid data.

Go vmihailenco/msgpack

Go's string type is a byte sequence with no UTF-8 guarantee
(similar to Pony). The library decodes str format values directly
into Go strings without UTF-8 validation. Go provides
utf8.Valid() in the stdlib for callers who want to validate
after decoding, but the msgpack library does not call it.

Summary

Library	Default decode behavior	Invalid UTF-8 handling
Python msgpack	UTF-8 decode to str	Error (configurable)
JS @msgpack/msgpack	UTF-8 decode to string	Replace (configurable)
Rust rmp	UTF-8 validated &str	Error; Raw type escape hatch
Go vmihailenco/msgpack	No validation	Caller's responsibility

Libraries in languages with distinct string/bytes types (Python,
Rust) enforce UTF-8 by default. Libraries in languages where
strings are byte arrays (Go) do not. Pony falls into the latter
category.

Design

Decision: opt-in validation

UTF-8 validation should be opt-in, not the default. Reasons:

1. Backward compatibility. The library has been returning
   unvalidated strings since its creation. Changing the default
   to validate would break existing callers that rely on the
   current behavior, including callers handling pre-2013 msgpack
   data where the same wire formats (now called str) carried
   raw bytes before the str/bin split was introduced.

2. Pony's type system does not distinguish validated strings.
   Unlike Rust, where &str and &[u8] are different types,
   Pony's String is always a byte array. Validation would be
   a runtime check with no compile-time representation of the
   result. The caller receives a String either way.

3. Performance. UTF-8 validation requires examining every
   byte of every decoded string. For workloads that decode many
   strings but do not require UTF-8 correctness (e.g., binary
   protocol keys that happen to be ASCII), mandatory validation
   is pure overhead.

4. Precedent. Go's msgpack library, the closest analog to
   Pony's situation, also does not validate by default.

Where validation lives

Validation is available at all decoder layers and the encoder:

• MessagePackDecoder: New methods alongside the existing
  str methods.
• MessagePackZeroCopyDecoder: Same new methods, mirroring
  the existing parity between the two decoder primitives.
• MessagePackStreamingDecoder: A configuration option that
  enables validation for all str values decoded via next().
• Encoder: New methods alongside the existing str methods.
• Public validator: A public primitive for caller-side
  validation in the "decode then validate" pattern.

Decoder API: validating str methods

Add a parallel set of str decoding methods that validate UTF-8
after reading the bytes. The naming convention appends _utf8 to
distinguish them from the existing non-validating methods.

For MessagePackDecoder (operates on buffered.Reader, returns
String iso^):

primitive MessagePackDecoder
  // Existing (unchanged):
  fun str(b: Reader): String iso^ ?
  fun fixstr(b: Reader): String iso^ ?
  fun str_8(b: Reader): String iso^ ?
  fun str_16(b: Reader): String iso^ ?
  fun str_32(b: Reader): String iso^ ?

  // New validating variants:
  fun str_utf8(b: Reader): String iso^ ?
  fun fixstr_utf8(b: Reader): String iso^ ?
  fun str_8_utf8(b: Reader): String iso^ ?
  fun str_16_utf8(b: Reader): String iso^ ?
  fun str_32_utf8(b: Reader): String iso^ ?

For MessagePackZeroCopyDecoder (operates on ZeroCopyReader,
returns String val):

primitive MessagePackZeroCopyDecoder
  // Existing (unchanged):
  fun str(b: ZeroCopyReader): String val ?
  fun fixstr(b: ZeroCopyReader): String val ?
  fun str_8(b: ZeroCopyReader): String val ?
  fun str_16(b: ZeroCopyReader): String val ?
  fun str_32(b: ZeroCopyReader): String val ?

  // New validating variants:
  fun str_utf8(b: ZeroCopyReader): String val ?
  fun fixstr_utf8(b: ZeroCopyReader): String val ?
  fun str_8_utf8(b: ZeroCopyReader): String val ?
  fun str_16_utf8(b: ZeroCopyReader): String val ?
  fun str_32_utf8(b: ZeroCopyReader): String val ?

Each _utf8 variant calls the corresponding existing method to
decode the string, then validates the result. If validation
fails, the method errors. This means the reader has already
consumed the bytes -- the caller cannot recover the raw bytes
after a validation failure through these methods.

The alternative -- returning a union like
(String iso^ | Array[U8] iso^) -- would change the return type
and complicate the common case where the caller just wants a
valid string or an error.

UTF-8 validation implementation

The validator uses String.utf32 to walk the string byte by
byte and detect invalid sequences. utf32 returns
(0xFFFD, 1) for invalid sequences -- the replacement character
with a consumed length of 1. A legitimately encoded U+FFFD
character (bytes 0xEF 0xBF 0xBD) returns (0xFFFD, 3). The
length field disambiguates.

The validator is a public primitive so callers can use the
"decode then validate" pattern (see "Accessing raw bytes on
decode failure" below):

primitive MessagePackValidateUTF8
  fun apply(s: String box): Bool =>
    var i: USize = 0
    while i < s.size() do
      try
        (let cp, let len) = s.utf32(i.isize())?
        if (cp == 0xFFFD) and (len == 1) then
          // Replacement character with length 1 means
          // the byte was not a valid UTF-8 sequence
          // start. But 0xEF 0xBF 0xBD is the actual
          // UTF-8 encoding of U+FFFD (length 3), so
          // length 1 distinguishes real replacement
          // characters from validation failures.
          return false
        end
        i = i + len.usize()
      else
        return false
      end
    end
    true

This implementation depends on String.utf32's error-reporting
convention. This is how Pony represents invalid UTF-8 throughout
the stdlib. The implementation should document this dependency
at the point of use.

Note: Pony's stdlib does not currently provide a built-in UTF-8
validation method on String. After this work is merged, a
follow-up issue will be opened to design an RFC for adding
UTF-8 validation to the stdlib's String type. If the stdlib
gains native validation in the future, this library should
migrate to use it and potentially deprecate
MessagePackValidateUTF8.

Streaming decoder configuration

The streaming decoder supports a configuration option for UTF-8
validation of str values. The constructor currently accepts a
limits parameter; validate_utf8 is added alongside it:

class MessagePackStreamingDecoder
  let _reader: ZeroCopyReader ref
  let _limits: MessagePackDecodeLimits val
  let _validate_utf8: Bool
  let _stack: Array[USize]

  new create(
    limits: MessagePackDecodeLimits val
      = MessagePackDecodeLimits,
    validate_utf8: Bool = false)
  =>
    _reader = ZeroCopyReader
    _limits = limits
    _validate_utf8 = validate_utf8
    _stack = Array[USize]

When _validate_utf8 is true, the _decode_fixstr and
_decode_str methods validate the decoded string before
returning it. These methods delegate to
MessagePackZeroCopyDecoder internally; validation occurs after
the zero-copy decoder returns the String val. On validation
failure, they return InvalidUtf8.

This is a constructor-time decision rather than per-call because
the streaming decoder already has a simple next(): DecodeResult
interface with no per-call options. Adding a parameter to next
would change the API for all callers. A constructor flag keeps
the interface clean.

Streaming decoder error type for UTF-8 failures

A new InvalidUtf8 primitive is added to the DecodeResult
union, distinct from InvalidData.

InvalidData means "the stream contains an invalid MessagePack
format byte -- the stream is corrupt and decoding should stop."
UTF-8 validation failure is categorically different: the
MessagePack framing is perfectly valid, only the string content
violates the spec's UTF-8 expectation. The stream is not corrupt
and decoding can safely continue.

Conflating these under InvalidData would force callers to use
out-of-band knowledge to distinguish two failure modes with
different recovery semantics. Distinct semantics deserve distinct
representations.

The updated DecodeResult union:

type DecodeResult is
  ( MessagePackValue
  | NotEnoughData
  | InvalidData
  | InvalidUtf8
  | LimitExceeded )

Encoder validation

The encoder adds _utf8 variants of the str encoding methods
that validate before writing, leaving the existing methods
unchanged:

primitive MessagePackEncoder
  // Existing (unchanged):
  fun str(b: Writer, v: ByteSeq) ?
  fun fixstr(b: Writer, v: ByteSeq) ?
  // ...

  // New validating variants:
  fun str_utf8(b: Writer, v: ByteSeq) ?
  fun fixstr_utf8(b: Writer, v: ByteSeq) ?
  // ...

The encoder methods accept ByteSeq, which is
(String val | Array[U8] val). The validation implementation
handles both cases: for String val, the validator uses
String.utf32 directly via MessagePackValidateUTF8; for
Array[U8] val, the bytes are wrapped in a String via
String.from_array before validation.

Accessing raw bytes on decode failure

The spec recommends that deserializers provide access to the
original byte array. The current non-validating methods already
satisfy this -- they return the raw bytes as a String. For
callers using the validating methods who also want the raw bytes
on failure, the pattern would be:

1. Try the _utf8 variant.
2. On failure, call the non-validating variant to get the raw
   bytes (but this requires re-reading from the buffer, which
   is not possible after the bytes have been consumed).

This is a limitation. Once a validating method errors, the bytes
are consumed and gone. The recommended mitigation is the "decode
then validate" pattern -- callers who need the raw bytes on
failure should use the non-validating method and then validate
the result themselves using MessagePackValidateUTF8:

// Decode without validation
let s: String val = MessagePackDecoder.str(reader)?

// Validate separately
if not MessagePackValidateUTF8(s) then
  // Handle invalid UTF-8: log, reject, use as raw bytes
end

This is the more Pony-idiomatic approach -- the caller controls
the flow and decides what to do with invalid data.

Follow-up

After this work is merged, open an issue on this repository to
track designing an RFC for adding UTF-8 validation to the Pony
stdlib's String type. UTF-8 validation is a general-purpose
need that does not belong permanently in a msgpack library. If
the stdlib gains String.is_valid_utf8 (or equivalent), this
library should migrate to use it.

Scope

This proposal covers:

• Validating _utf8 variants of str decoding methods on
  MessagePackDecoder
• Validating _utf8 variants of str decoding methods on
  MessagePackZeroCopyDecoder
• A validate_utf8 constructor option on
  MessagePackStreamingDecoder
• A new InvalidUtf8 primitive in the DecodeResult union
• Validating _utf8 variants of str encoding methods on
  MessagePackEncoder
• A public MessagePackValidateUTF8 primitive for caller-side
  validation

This proposal does not cover:

• Changing the default behavior of existing methods (the current
  non-validating behavior is preserved)
• A validated string type (Pony's type system does not support
  this without runtime wrappers)
• Pre-2013 raw format compatibility (the library already has
  bin format methods for raw byte data)
• Replacement character substitution (replacing invalid bytes
  with U+FFFD rather than erroring) -- this could be a separate
  future enhancement

[SeanTAllen]

Design decisions resolved

The three open design decisions have been resolved and the document updated to reflect them:

1. Streaming decoder error type: New InvalidUtf8 primitive added to DecodeResult, distinct from InvalidData. The two have different recovery semantics — InvalidData means corrupt framing (stop decoding), InvalidUtf8 means valid framing with invalid string content (can continue). Distinct semantics deserve distinct representations.

2. Validation implementation: Use String.utf32 to walk the string and detect invalid sequences. Simpler than a standalone byte-level validator, and the (0xFFFD, 1) error-reporting convention is how Pony represents invalid UTF-8 throughout the stdlib. The dependency should be documented at point of use.

3. Validation primitive visibility: Public MessagePackValidateUTF8 primitive. This enables the "decode then validate" pattern — the only way callers can get both raw bytes and validation, since the _utf8 methods consume bytes before erroring. Without a public validator, callers who want this pattern would have to write their own.

Additionally, a follow-up section was added: after this work merges, an issue will be opened to track designing an RFC for adding UTF-8 validation to the Pony stdlib's String type.

[SeanTAllen]

implemented in #76