TOON spec issues-- per Fable

[liquidaty]

In the course of implementing a toon parsing library (liquidaty/yatl), I had a question about how the spec mandates whitespace trimming (just ascii vs all UTF8), which led me to look at the reference implementation, which eventually led me to ask Fable to compare the spec to the reference implementation (and to my implementation, though that topic is omitted from the rest of this post) and to identify any areas of either inconsistency or ambiguity in the spec. It came up with the below.

I have reviewed the report and it's a bit more nit-picky than I personally would be, and I might flip some of its recommendations to go the other way.

Anyway, I would be happy to log these each as issues if this repo owners would like, just not sure if that is encouraged or frowned upon when the analysis was generated by AI. From my perspective, that shouldn't matter, because if it's going to help the spec to be more AI-ergonomic then that's probably a good thing for everyone. But, I've seen some unexpectedly (to me at least) strong get-anything-AI-generated-out-of-here sentiment floating around (not specifically here, just in general) so thought I'd post here for discussion first.

---

Feedback on the TOON Specification: Decoder Ambiguities and Reference-Implementation Divergences

This document collects constructive feedback on the TOON specification
(toon-format/spec), based on an independent implementation effort that compared
the specification text, the reference TypeScript implementation
(@toon-format/toon, v2.1.1), and observed decoder behavior on edge-case
inputs in strict mode. Every behavior cited below was verified by executing the
reference implementation.

The issues fall into three groups: (A) terms the specification uses but never
defines, (B) decoder behavior the specification leaves unspecified, where
independent implementations will predictably diverge, and (C) places where the
reference implementation and the specification text disagree, so one of the two
should be amended. Suggested conformance fixtures are included so each
resolution can be pinned by the language-agnostic test suite under tests/.

---

1. "Space" vs. "whitespace" is undefined, and token trimming diverges (A + C)

Spec text. §11.2 says of splitting: "surrounding spaces are trimmed";
Appendix B.3 repeats "Trim surrounding spaces around each token." Where the
specification is precise, "space" means U+0020 (§12: "exactly one ASCII space
(U+0020)"). But three passages use the undefined word "whitespace" instead:
§9.1 ("empty tokens (including those surrounded by whitespace)"), §12
("Surrounding whitespace around tokens SHOULD be tolerated"), and §7.2 ("It has
leading or trailing whitespace"). The specification never defines either term.

Reference implementation. parseDelimitedValues trims each split token
with JavaScript's String.prototype.trim(), which removes the full ECMAScript
whitespace set — HTAB, NBSP (U+00A0), U+2000–U+200A, U+3000, U+FEFF, and line
terminators — not just U+0020. Verified:

• x[2]: a,<U+00A0>b decodes to ["a","b"] — the NBSP is silently deleted.
• x[2]: a,<TAB>b decodes to ["a","b"] — the tab is silently deleted.

Why it matters. Under a literal reading of §11.2 ("spaces"), a bare token
may begin with NBSP, and §7.2 (if "whitespace" there is read as ASCII-only)
does not force an encoder to quote it. A decoder that trims Unicode whitespace
then corrupts round-trips. HTAB is worse: it can itself be the active
delimiter (§6), so any rule that trims tabs from tokens must be carefully
scoped to non-tab-delimited documents — a subtlety the current trim()-based
behavior papers over. Independent implementations reading "spaces" literally
will disagree with the reference implementation on the same inputs.

Recommendation.

1. Add "space" and "whitespace" to §1 (Terminology). Suggested: space =
   U+0020 only; use "whitespace" nowhere normative, or define it explicitly.
2. State in §11.2 that trimming removes U+0020 only (this matches B.3 and
   makes tab-delimited documents unambiguous), and align the reference
   implementation (replace trim() with a U+0020-only trim).
3. Re-word §7.2's "leading or trailing whitespace" to name the exact
   codepoints that trigger mandatory quoting.

Suggested fixtures. x[2]: a, b (expect ["a"," b"]);
x[2\t]: a\tb with spaces padding the tokens; x[2]: a,\tb in a
comma-delimited document (expected outcome per resolution of §7.2 wording).

---

2. Decoder handling of keys that violate the key grammar is unspecified (B)

Spec text. §7.3 gives an encoder rule: keys MAY be unquoted only if they
match ^[A-Za-z_][A-Za-z0-9_.]*$, otherwise they MUST be quoted. Nothing in
§4, §7.4, or the §14 strict-mode checklist says what a decoder must do with a
line such as a b: 1 or a : 1. The answer also feeds §5 root-form
discovery: whether a b: 1 is a "key-value line" determines whether the
document is an object or an (invalid) root primitive.

Reference implementation. Lenient: everything before the first unquoted
colon is taken as the key. a b: 1 → {"a b": 1}; a : 1 → {"a": 1}
(the space before the colon is silently dropped).

Recommendation. Specify one of:

• Lenient: decoders MUST accept any non-empty text before the colon as a
  literal key (and state whether it is trimmed), and define "key-value line"
  in §5 accordingly; or
• Strict: in strict mode, an unquoted key that does not match the §7.3
  pattern MUST error (add it to §14.2).

Either answer is workable; today each implementer must guess, and the guesses
will differ. If leniency is chosen, note the §7.3 grammar remains an encoder
MUST.

Suggested fixtures. a b: 1, a : 1, -x: 1, "a b": 1 (control).

---

3. May a strict decoder reject unquoted values an encoder was required to quote? (B)

Spec text. §7.2 and §15 say strings containing colons (among others) MUST
be quoted — an encoder rule. §4 says unquoted tokens that are not
booleans/null/numbers/[] decode "otherwise → string". §14 is titled an
"Authoritative Checklist" of strict-mode errors, and rejecting such tokens is
not on it — but the section does not say whether the list is exhaustive
(i.e., whether a conforming strict decoder may add errors).

Reference implementation. Lenient: x: a:b decodes to {"x": "a:b"}.

Why it matters. A decoder that validates "would a conforming encoder have
produced this?" (a natural strict-mode posture, and a useful one for detecting
truncation/injection per §15) will reject x: a:b while the reference
implementation accepts it. Whether that decoder is conforming is currently
unanswerable.

Recommendation. Add one sentence to §14: either "decoders MUST NOT raise
strict-mode errors beyond this list" (making §4's fall-through binding), or
"decoders MAY additionally reject documents no conforming encoder can emit,
provided such rejections are documented." The first maximizes interop; the
second legitimizes validating decoders. Please also state which reading the
reference implementation follows.

Suggested fixtures. x: a:b, x: [not-a-header, x: "a" b
(the last MUST error per §7.1's unterminated/trailing rule — worth pinning).

---

4. key[] (bracket segment with no length, no colon) — §5 and §6 pull in opposite directions (B)

Spec text. §6 requires every array header to contain a length and end in a
colon; §4 says a key without a colon MUST error; §5 root-form discovery says a
single non-empty line that is neither a valid header nor a key-value line
decodes as a root primitive; and §7.2 says strings containing brackets MUST be
quoted (encoder side).

Reference implementation. key[] alone in a document decodes to the root
primitive string "key[]".

Why it matters. Three plausible outcomes — error as a malformed header,
error as a key without a colon, or fall through to a root scalar — and the
specification supports each through a different section. This is exactly the
kind of input fuzzers find immediately.

Recommendation. State explicitly in §5 or §6 which interpretation wins for
a line that begins like a header (bare-key [ …) but fails header syntax:
hard error in strict mode, or fall-through to key-value/primitive parsing.
(§14.2 already does this for one sub-case — content between the bracket
segment and the colon; extending that clause to all malformed-header shapes
would close the gap.)

Suggested fixtures. key[], key[2] (no colon), key[]: x, [2] extra.

---

5. - [] as a list item: the reference implementation accepts a form §9.2 excludes (C)

Spec text. §9.2 says empty inner arrays in list position are written
- [0<delim?>]: and adds: "The key: [] field-level form (§9.1) does NOT
apply to list-item inner arrays." §9.1 enumerates the only positions where the
bare [] token is meaningful (root document, key: []).

Reference implementation. x[1]: followed by - [] decodes to
{"x": [[]]} — the excluded form is accepted.

Recommendation. Pick one and make both artifacts agree: either amend §9.2
to permit - [] as a decoder leniency (and say whether strict mode allows
it), or keep the exclusion and fix the reference implementation to error.
Given §9.2's current MUST-style language, the smaller change is a decoder
error plus a negative fixture.

Suggested fixtures. x[1]:\n - [] (expected: error, or [[]] per
resolution); x[1]:\n - [0]: (control, expect [[]]).

---

6. Out-of-range numbers: the option list omits "error", and the reference implementation's fallback is undocumented (B + C)

Spec text. §4: if a numeric token is not representable, an implementation
"MAY return a higher-precision numeric type, return a string, or return an
approximate numeric value if that is the documented policy. Implementations
MUST document their out-of-range policy; lossless-first is RECOMMENDED…"

Reference implementation. Two verified behaviors:

• x: 1e309 (overflows IEEE-754 double) decodes to the string "1e309" —
  a silent type change from number to string.
• x: 9007199254740993 (2⁵³ + 1) decodes to 9007199254740992 — silent
  precision loss, i.e. the approximate policy, not the RECOMMENDED
  lossless-first one.

As far as I could find, neither policy is documented where §4 requires it.

Why it matters. A value silently changing type (number → string) or
magnitude (…993 → …992) is the most dangerous failure mode for a data
interchange format: nothing errors, and the corruption surfaces far
downstream. Implementations in languages with 64-bit integers or big-number
types will also disagree with the reference implementation on the same
documents.

Recommendation.

1. Add "raise an error (RECOMMENDED default in strict mode)" to §4's list of
   permitted out-of-range behaviors — for validating decoders it is the only
   honest option, and today it is arguably non-conforming because the MAY-list
   omits it.
2. Document the reference implementation's actual policy (string fallback on
   overflow; double approximation above 2⁵³), per §4's own MUST.
3. Consider a note that integers with |n| > 2⁵³ are not round-trippable
   through IEEE-754-based implementations, so encoders targeting maximal
   interop SHOULD quote them.

Suggested fixtures. x: 1e309, x: -1e309, x: 9007199254740993,
x: 0.1000000000000000055511151231257827 — each with the expected value and
permitted alternates spelled out.

---

7. Invalid UTF-8 input is unaddressed (B)

Spec text. §17 fixes the charset to UTF-8; §15 says decoders SHOULD accept
valid UTF-8. Nothing says what a decoder must do with an invalid byte
sequence (overlong encodings, truncated sequences, stray continuation bytes,
lone-surrogate encodings in raw bytes).

Reference implementation. Structurally silent on this: decode() takes a
JavaScript string, so byte-level validation happens — or silently does not —
at the caller's I/O layer, which typically substitutes U+FFFD. Malformed input
can therefore be laundered into a "successfully" decoded document.

Recommendation. Add a sentence to §15 or §17: decoders that accept byte
input MUST (strict) / SHOULD (non-strict) reject invalid UTF-8, and
string-based APIs SHOULD document that byte validation is the caller's
responsibility. A byte-level fixture file in tests/ (invalid sequences that
MUST error for byte-accepting decoders) would make this testable.

---

8. Smaller editorial notes

• §14's exhaustiveness (see issue 3) also affects §12's "Surrounding
  whitespace around tokens SHOULD be tolerated" — as written, a strict decoder
  that does not tolerate it is technically conforming (SHOULD), while one
  that tolerates more than U+0020 corrupts data (issue 1). Tightening issue 1
  resolves this clause too.
• §7.1 surrogate handling is a bright spot: the table's "MUST reject
  surrogate \uXXXX escapes, including pairs" rule is unambiguous, and the
  reference implementation follows it exactly (verified: the escaped pair
  "\uD83D\uDE00" is rejected with a clear message, while the same character
  as literal UTF-8 is accepted). The escape-handling section would be a good
  template for specifying the areas above.
• The reference implementation's strict-mode diagnostics (duplicate keys,
  indentation multiples, row/count mismatches) match §14 precisely in every
  case tested; the divergences reported here are confined to the areas where
  the specification is silent or self-contradictory.

---

All reference-implementation behaviors were verified against
@toon-format/toon v2.1.1 (npm) in default strict mode. Happy to contribute
any of the suggested fixtures to tests/ as a PR if that is useful.