feat(jsonb): Support raw JSON body literals under JSONB via SAX replay

`StreamingJsonbBuilder::add_raw` was a no-op stub; raw string bodies (`req.body = R"({"temp":22.5})"`) were a compile error under `NOTE_JSONB`. With the SAX-events-in refactor in place, the lexer can now drive any `JsonBuilder` — so the natural fix is to SAX-parse the fragment and replay the events as JSONB opcodes.

New nested `ReplaySink : JsonSink` (private to `StreamingJsonbBuilder`, gets access to the private emit_* helpers). Tracks object/array context with a 32-bit nesting stack and re-emits opcodes:

- on_object_begin/on_array_begin: kItem(outer_key) at depth 0, then
kItem(key) for object members at depth > 0, then the begin opcode
- on_object_end/on_array_end: matching end opcode, pop context
- on_bool/on_string/on_null/on_number: kItem only when in an object;
on_number sniffs `.`/`e`/`E` to pick kInt32 vs kDouble

Re-enable the raw-string `BodyValue` ctor under JSONB by changing the gate from `#if NOTE_JSONB` to `#if NOTE_JSONB && NOTE_MINIMAL`. The GCC-14+ consteval validation path still applies, so on the supported toolchain a malformed literal is rejected at compile time the same way it is under JSON.

The `NOTE_MINIMAL` carve-out: a JSONB+MINIMAL build can't afford the ~6 KB the SAX parser + parse_double + snprintf-%g pull in via the vtable, so under that combination `add_raw` stays a no-op and the BodyValue ctor stays disabled (avoids silent data loss). AVR (NOTE_MINIMAL=1) flash holds at 27,886 B — unchanged from baseline.

Tests (`tests/test_jsonb.cpp`, gated `#if !NOTE_MINIMAL`):

- flat object with bool field (full byte-by-byte assertion)
- mixed primitives (int / float / string / null) with cursor walk
- nested object
- primitive array (verifies no kItem between array elements)
- empty fragment short-circuit

`tests/compile_fail/jsonb_raw_body.cpp` moves to `tests/compile_check/jsonb_raw_body.cpp` and flips polarity (raw body now compiles, guarded `#if !NOTE_MINIMAL`).

`tests/compile_check/jsonb_body_alternatives.cpp` extended with two runtime-parameter variants (lambda capture and struct construction) to show how dynamic data enters a body — raw string literals stay the compile-time-fixed path; the lambda/struct shapes are for runtime values.

Docs updated:
- `docs/jsonb.md` Limitations bullet rewritten around the lexer cost
and the NOTE_MINIMAL carve-out.
- `docs/known-issues.md` "JSONB raw bodies" section reframed as the
NOTE_MINIMAL caveat; closes "resolved outside MINIMAL" loop.
- `docs/troubleshooting.md` See-also bullet drops "JSONB raw bodies"
from the top-level known-issues list.

Verified host build (2031 cases / 7704 assertions pass) and AVR build (27,886 / 782, baseline match).

Author: m-mcgowan

docs/jsonb.md

@@ -87,7 +87,7 @@ JSONB does not use CRC. The `"crc":"XXXX:YYYYYYYY"` field is a JSON-specific mec
 
 ## Limitations
 
-- **No raw JSON embedding:** `add_raw()` is a no-op in JSONB mode. Raw JSON fragments (e.g., body lambdas that emit pre-formatted JSON) cannot be embedded in JSONB without conversion.
+- **Raw JSON embedding pulls the SAX lexer in:** `add_raw()` works under JSONB — it SAX-parses the fragment and re-emits opcodes — but the lexer must be linked in (~6 KB of full-text parser + float printf). Under `NOTE_MINIMAL` the impl is gated off and the `BodyValue` raw-string constructor is disabled, so raw bodies are a compile error on the AVR profile. Use `body(lambda)` or `body(struct)` instead; those shapes never go through `add_raw`. See [known-issues.md](known-issues.md) for the carve-out.
 - **Compile-time only:** The wire format is selected at compile time via `NOTE_JSONB`. Runtime switching is not supported.
 - **Firmware requirement:** Requires Notecard firmware 11.x or later. Earlier firmware does not support JSONB.
 

docs/known-issues.md

@@ -37,28 +37,31 @@ with the fix. The `#if !defined(__clang__)` guards can be removed once
 the bug is resolved. The compile-fail tests have `#error` skips for
 Clang so they'll start failing (correctly) when the fix lands.
 
-## JSONB: raw JSON string bodies are a compile error
+## JSONB + NOTE_MINIMAL: raw JSON string bodies remain a compile error
 
-**Affects:** builds with `NOTE_JSONB=1` (including `NOTE_MINIMAL`).
+**Affects:** builds with `NOTE_JSONB=1` *and* `NOTE_MINIMAL=1` (the
+AVR / 32 KB-flash profile).
 
-**Symptom:** `req.body = R"({"temp":22.5})"` or `req.body("json string")`
-fails to compile with "no viable overloaded '='".
+**Symptom:** `req.body = R"({"temp":22.5})"` fails with "no viable
+overloaded `=`" on the AVR build.
 
-**Cause:** JSONB cannot embed raw JSON text fragments. The `add_raw()`
-builder method is a no-op in JSONB mode, so the raw-string `BodyValue`
-constructors are disabled to prevent silent data loss.
+**Cause:** The general `add_raw()` implementation under JSONB
+SAX-parses the fragment and replays the events as opcodes. That pulls
+~6 KB of full-text JSON parser, `parse_double`, and `snprintf("%g", …)`
+into the binary — more than the ATmega328P's 32 KB flash budget can
+absorb. Under `NOTE_MINIMAL` the impl is therefore a no-op, and the
+`BodyValue` raw-string constructor is correspondingly disabled to
+prevent silent data loss.
 
-**Workaround:** Use a builder lambda or typed struct instead:
+**Workaround:** Use a builder lambda or typed struct — both shapes
+never go through `add_raw` and so don't pull the lexer in:
 
 ```cpp
-// Lambda body — works with both JSON and JSONB
-req.body(note::body([](note::JsonBuilder& b) {
-    b.add("temp", 22.5);
-}));
-
-// Typed struct body — works with both JSON and JSONB
+req.body(note::body([](note::JsonBuilder& b) { b.add("temp", 22.5); }));
 req.body(Readings{.temperature = 22.5, .humidity = 60});
 ```
 
-A compile-fail test (`tests/compile_fail/jsonb_raw_body.cpp`) verifies
-this behavior.
+**Resolved at runtime under `NOTE_JSONB && !NOTE_MINIMAL`:** non-AVR
+JSONB builds support `req.body = R"(...)"` directly — the same surface
+as under JSON. See `tests/compile_check/jsonb_raw_body.cpp` and the
+`add_raw` cases in `tests/test_jsonb.cpp`.

docs/troubleshooting.md

@@ -140,6 +140,6 @@ Fix: enable wire tracing (`NOTE_DEBUG_ENABLED=1` plus a listener that dumps both
 
 ## See also
 
-- [`known-issues.md`](known-issues.md) — confirmed library bugs with workarounds (Apple Clang `consteval`, JSONB raw bodies)
+- [`known-issues.md`](known-issues.md) — confirmed library bugs with workarounds (Apple Clang `consteval`)
 - [`debugging.md`](debugging.md) — wire tracing, transport diagnostics, debug categories
 - [`getting-started.md`](getting-started.md) — top-down walkthrough from a clean project to your first request

src/note/body.hpp

@@ -50,9 +50,12 @@ class BodyValue {
 
     constexpr BodyValue() = default;
 
-#if NOTE_JSONB
-    // Raw JSON string bodies are not supported with JSONB wire format.
-    // Use body() with a lambda or typed struct instead.
+#if NOTE_JSONB && NOTE_MINIMAL
+    // Raw JSON string bodies are unavailable under the JSONB + MINIMAL
+    // combination: the SAX-replay add_raw impl (jsonb.hpp) would pull
+    // ~6 KB of full-text parser + float printf into the AVR build, which
+    // doesn't fit. Use body() with a lambda or typed struct instead —
+    // those shapes never go through add_raw. See docs/known-issues.md.
 #elif __cplusplus >= 202002L && !defined(__clang__) && __GNUC__ >= 14
     // String literal: validated at compile time as well-formed JSON object.
     //
@@ -62,6 +65,11 @@ class BodyValue {
     // expression" on GCC 13.x — including 13.4, even though the direct call
     // form (`constexpr BodyValue v = "..."`) compiles cleanly. Stay on the
     // GCC 14+ gate until the inherited-ctor case works empirically.
+    //
+    // Wire format: in JSON the fragment is embedded verbatim via
+    // `add_raw`; in JSONB it's SAX-parsed and re-encoded as opcodes by
+    // `StreamingJsonbBuilder::add_raw`. Either way the user-facing
+    // surface is the same.
     template<std::size_t N>
     consteval BodyValue(const char (&s)[N])
         : str_(string_view(s, N - 1)), write_fn_(&write_string) {

src/note/jsonb.hpp

@@ -11,6 +11,7 @@
 /// transport path).
 
 #include "json.hpp"
+#include "json_sax.hpp"
 #include "json_sax_streaming.hpp"
 #include "lexer/sax_adapter.hpp"
 #include "link/cobs.hpp"
@@ -105,10 +106,27 @@ class StreamingJsonbBuilder : public JsonBuilder {
         return *this;
     }
 
-    JsonBuilder& add_raw(string_view, string_view) override {
-        // No-op — raw JSON fragments cannot be embedded in JSONB.
-        // BodyValue's raw-string constructor is disabled when NOTE_JSONB=1
-        // so this path is unreachable in normal use.
+    JsonBuilder& add_raw(string_view key, string_view json_fragment) override {
+#if !NOTE_MINIMAL
+        // Re-encode the raw JSON fragment as JSONB opcodes by SAX-parsing
+        // it through ReplaySink (defined below). The fragment must be a
+        // JSON object — sax_parse rejects other shapes — so a body like
+        // `R"({"temp":22.5})"` lands as `kItem(key) + kBeginObject + ...
+        // + kEndObject` in the wire stream.
+        //
+        // Gated off under NOTE_MINIMAL so the AVR/JSONB build doesn't
+        // pull the full-text SAX parser (sax_parse, parse_double,
+        // snprintf-%g) into the binary — that combination is ~6 KB of
+        // flash that the 32 KB Uno target can't absorb. The BodyValue
+        // raw-string constructor is correspondingly disabled in body.hpp
+        // under the same combination, so callers can't reach this code
+        // path with a no-op underneath.
+        if (json_fragment.empty()) return *this;
+        ReplaySink sink(*this, key);
+        (void)sax_parse(json_fragment, sink);
+#else
+        (void)key; (void)json_fragment;
+#endif
         return *this;
     }
 
@@ -182,6 +200,89 @@ class StreamingJsonbBuilder : public JsonBuilder {
     JsonWriter& writer_;
     bool closed_ = false;
 
+    // SAX → JSONB-opcode replay. Used by add_raw to translate a raw JSON
+    // fragment (e.g. a body literal) into JSONB opcodes one event at a
+    // time. Nested so it can reach the private emit_* helpers directly.
+    //
+    // Context tracking: a single uint32_t stack records "are we in an
+    // array?" per nesting level (1 bit per level, LSB = current). 32
+    // levels is far beyond the depth of any Notecard body in practice.
+    struct ReplaySink : JsonSink {
+        StreamingJsonbBuilder& b;
+        string_view outer_key;
+        uint32_t array_stack = 0;
+        int depth = 0;
+
+        ReplaySink(StreamingJsonbBuilder& bb, string_view k) : b(bb), outer_key(k) {}
+
+        void push(bool is_arr) { array_stack = (array_stack << 1) | (is_arr ? 1u : 0u); }
+        void pop()             { array_stack >>= 1; }
+        bool in_array() const  { return (array_stack & 1u) != 0; }
+
+        // Emit kItem(k) when the current value is a field of an object.
+        // Skipped for array elements (no kItem) and for the root (the
+        // outer_key was emitted by the caller via on_object_begin /
+        // on_array_begin at depth 0).
+        void emit_key(string_view k) {
+            if (depth > 0 && !in_array()) b.emit_item(k);
+        }
+
+        void on_object_begin(string_view k) override {
+            if (depth == 0)         b.emit_item(outer_key);
+            else if (!in_array())   b.emit_item(k);
+            b.emit(jsonb::kBeginObject);
+            push(false);
+            depth++;
+        }
+        void on_object_end(string_view) override {
+            b.emit(jsonb::kEndObject);
+            depth--;
+            pop();
+        }
+        void on_array_begin(string_view k) override {
+            if (depth == 0)         b.emit_item(outer_key);
+            else if (!in_array())   b.emit_item(k);
+            b.emit(jsonb::kBeginArray);
+            push(true);
+            depth++;
+        }
+        void on_array_end(string_view) override {
+            b.emit(jsonb::kEndArray);
+            depth--;
+            pop();
+        }
+        void on_null(string_view k) override {
+            emit_key(k);
+            b.emit(jsonb::kNull);
+        }
+        void on_bool(string_view k, bool v) override {
+            emit_key(k);
+            b.emit(v ? jsonb::kTrue : jsonb::kFalse);
+        }
+        void on_string(string_view k, string_view v) override {
+            emit_key(k);
+            b.emit(jsonb::kString);
+            b.writer_.write(v.data(), v.size());
+            b.emit('\0');
+        }
+        void on_number(string_view k, string_view raw) override {
+            emit_key(k);
+            bool is_float = false;
+            for (char c : raw) {
+                if (c == '.' || c == 'e' || c == 'E') { is_float = true; break; }
+            }
+            if (is_float) {
+                double d = note::parse_double(raw);
+                b.emit(jsonb::kDouble);
+                b.emit_bytes(&d, 8);
+            } else {
+                json_int_t i = note::parse_int(raw);
+                b.emit(jsonb::kInt32);
+                b.emit_le32(static_cast<int32_t>(i));
+            }
+        }
+    };
+
     void emit(uint8_t opcode) {
         writer_.write(reinterpret_cast<const char*>(&opcode), 1);
     }

tests/compile_check/README.md

@@ -28,10 +28,12 @@ Each file is self-contained and compiled exactly once:
 - `field_types.cpp` — instantiates every generated field type to
   catch `ResponseField` template instantiation errors that might
   otherwise only appear per-consumer.
-- `jsonb_body_alternatives.cpp` — under `NOTE_JSONB=1`, raw string
-  bodies are a compile error (intentional). This file compiles the
-  *supported* body-setter shapes (`body(...)` lambdas, typed
-  structs) to ensure they still work.
+- `jsonb_body_alternatives.cpp` — under `NOTE_JSONB=1`, compiles the
+  `body(...)` lambda and typed-struct shapes, which never go through
+  `add_raw` and so avoid pulling the SAX lexer into the build.
+- `jsonb_raw_body.cpp` — under `NOTE_JSONB=1`, raw string body
+  literals (`req.body = R"(...)"`) compile and produce JSONB opcodes
+  via `add_raw`'s SAX-replay path (skipped under `NOTE_MINIMAL`).
 - `minimal_mode.cpp` — `-DNOTE_MINIMAL=1` full build-check. Most AVR
   targets use this flag combination; this guards against a compile-
   time regression.

tests/compile_check/jsonb_body_alternatives.cpp

@@ -1,16 +1,29 @@
 // Compile-check: lambda and typed struct bodies work under NOTE_JSONB.
-// These are the recommended alternatives to raw JSON string bodies.
+// These shapes never go through add_raw, so they avoid pulling the SAX
+// lexer into the build path that raw string bodies use.
+//
+// Each shape is shown twice: once with literal values, and once with
+// runtime-supplied values, to demonstrate how dynamic data enters the
+// body. Raw string bodies (`req.body = R"(...)"`) carry no runtime
+// substitution — they're for compile-time-fixed JSON only; reach for
+// the lambda or struct shape when any field varies at runtime.
 #define NOTE_JSONB 1
 #include <note/api/note_add.hpp>
 #include <note/body.hpp>
 
+#include <cstdint>
+#include <cstdlib>   // std::rand
+
 struct Readings {
     float temperature;
     int32_t humidity;
-    NOTE_FIELDS(temperature, humidity)
+    int32_t sequence;
+    NOTE_FIELDS(temperature, humidity, sequence)
 };
 
-void test_lambda_body() {
+// Static lambda body — equivalent to a raw string literal, but without
+// the SAX-replay cost on JSONB builds.
+void test_lambda_body_static() {
     note::api::NoteAdd req;
     req.file = "sensors.qo";
     req.body(note::body([](note::JsonBuilder& b) {
@@ -19,8 +32,38 @@ void test_lambda_body() {
     }));
 }
 
-void test_typed_struct_body() {
+// Lambda body with runtime values — the closure captures by reference,
+// so each field can be a runtime expression. This is the shape to
+// reach for when "build a JSON body with a random integer" or any
+// other dynamic data needs to land on the wire.
+void test_lambda_body_runtime(float measured_temp, int32_t measured_humidity) {
+    int32_t sequence = std::rand();
+    note::api::NoteAdd req;
+    req.file = "sensors.qo";
+    req.body(note::body([&](note::JsonBuilder& b) {
+        b.add("temperature", static_cast<double>(measured_temp));
+        b.add("humidity", measured_humidity);
+        b.add("sequence", sequence);
+    }));
+}
+
+// Typed-struct body, fixed values.
+void test_typed_struct_body_static() {
+    note::api::NoteAdd req;
+    req.file = "sensors.qo";
+    req.body(Readings{22.5f, 60, 0});
+}
+
+// Typed-struct body with runtime values — the struct is built normally
+// in C++ and serialised field-by-field via StructSink. NOTE_FIELDS
+// enumerates the fields the library should walk.
+void test_typed_struct_body_runtime(float measured_temp, int32_t measured_humidity) {
+    Readings r{
+        .temperature = measured_temp,
+        .humidity    = measured_humidity,
+        .sequence    = std::rand(),
+    };
     note::api::NoteAdd req;
     req.file = "sensors.qo";
-    req.body(Readings{22.5f, 60});
+    req.body(r);
 }

tests/compile_check/jsonb_raw_body.cpp

@@ -0,0 +1,12 @@
+// Compile-check: raw JSON string body literals compile under NOTE_JSONB
+// (when NOTE_MINIMAL is not set — see docs/known-issues.md for the AVR
+// carve-out). The JSONB builder SAX-parses the literal and re-encodes
+// it as opcodes, so the same surface that works under JSON works here.
+#define NOTE_JSONB 1
+#include <note/api/note_add.hpp>
+#if !NOTE_MINIMAL
+void test() {
+    note::api::NoteAdd req;
+    req.body = R"({"temp":22.5})";  // expected: compiles cleanly
+}
+#endif

tests/compile_fail/README.md

@@ -58,8 +58,6 @@ Some backend modes forbid certain input shapes:
 - `buffered_requires_string.cpp`, `debug_requires_string.cpp` —
   callbacks that take `string_view` can't accept `const char*` on
   certain configurations.
-- `jsonb_raw_body.cpp` — raw JSON string body literals are forbidden
-  when `NOTE_JSONB=1`.
 
 ### Generic API guards