feat: lazy table cjson-compatible API (qd.decode / qd.encode)

[membphis]

Summary

Adds a cjson-shaped API on top of quickdecode so callers can migrate from cjson with two symbol swaps (cjson → qd, cjson.encode → qd.encode) while keeping quickdecode's lazy-evaluation win for "parse + extract a few fields" workloads.

• qd.decode(json) returns a Lua table with LazyObject / LazyArray metatable; reads go through __index to FFI on demand.
• qd.encode(v) works on lazy proxies (original-substring fast path), real Lua tables (matching cjson.encode output), and mixed trees. Nested mutations propagate correctly.
• qd.materialize(v) recursively converts a lazy view into a plain Lua table for callers that have to pass to a third-party encoder.
• qd.pairs / qd.ipairs, __pairs / __ipairs (LJ52), __len, __newindex (shallow first-write materialization).
• qd.null / qd.empty_array_mt aliased to cjson.null / cjson.empty_array_mt when cjson is loaded, with local fallbacks otherwise.

Two new Rust FFI exports support this:

• qjd_cursor_bytes — original-buffer byte range, for substring emit on unmodified subtrees.
• qjd_cursor_object_entry_at — i-th key/value pair of an object cursor, for iterators and materialization.

Plus a pre-existing Rust correctness fix surfaced during implementation: walk_children in src/cursor.rs previously skipped the trailing scalar element of arrays (while i < end → while i <= end with empty-container guard). Direct regression test in tests/ffi_cursor.rs.

Why

The library's headline pitch is 14–44× faster than cjson on "parse once, read a few fields" workloads, but its existing path-based API (d:get_str("messages[0].role")) requires rewriting every cjson-shaped call site to migrate. The lazy table API makes the migration largely a two-symbol swap while preserving the underlying lazy advantage (untouched subtrees never decode, encode emits the original bytes via memcpy).

Bench results vs cjson (median ops/s):

payload	cjson.decode + access 3	qd.decode + t.field x3	qd.decode + qd.encode (unmodified)
100 KB	~4,000	~55,000 (14×)	~62,000 (16×)
1 MB	~340	~7,400 (22×)	~9,800 (29×)

The lazy-table read path is ~30–40% behind the existing path-API on small payloads (per-__index dispatch + transient cdata wrap), converging at multi-MB sizes. Documented in README Roadmap as a structural gap worth investigating if a workload need surfaces.

Architecture

• lua/quickdecode/table.lua (new, 480 lines) — LazyObject / LazyArray metatables, qd.decode/encode/materialize/pairs/ipairs, sentinel bridging.
• lua/quickdecode.lua — adds two ffi.cdef lines for the new exports, re-exports the lazy API at the top level (so qd.decode works without separately requiring quickdecode.table).
• src/ffi.rs, src/doc.rs, src/cursor.rs, include/lua_quick_decode.h — 2 new FFI exports + the walk_children fix.
• tests/ffi_cursor_bytes.rs, tests/ffi_object_iter.rs (new) — direct FFI coverage.
• tests/lua/lazy_table_spec.lua (new, 332 lines) — 75 busted tests including cjson round-trip equivalence over 6 fixtures + sentinel coverage + nested-mutation + cached-proxy identity regression tests.
• benches/lua_bench.lua — two new rows (qd.decode + t.field x3, qd.decode + qd.encode (unmodified)) in every scenario + the interleaved block.
• docs/superpowers/specs/2026-05-16-lazy-table-cjson-compat-design.md — design doc.
• docs/superpowers/plans/2026-05-16-lazy-table-cjson-compat.md — implementation plan.

Notable correctness work

• LuaJIT FFI cdata aliasing — Discovered during Task 4: box[0] on a qjd_cursor[1] returns a reference, not a copy, so storing it in a Lua table and then reusing the box for the next FFI call silently corrupts the stored reference. Solved with a per-view own_box pattern (wrap_child does ffi.copy from a passed-in source box into a fresh allocation, kept alive on the view as _cur_box).
• walk_children trailing scalar — Pre-existing latent bug, found via the t[4] == qt.null test on [10,"x",true,null]. Fixed and locked in by a Rust unit test (tests/ffi_cursor.rs::walk_children_trailing_scalar_*).
• Nested-mutation encode — Caught in final review: t.a.b.c = 999 materializes only t.a.b, but t's metatable is still LazyObject. Plain substring fast path emits the original (unmutated) bytes. Fixed with an is_dirty(v) walker that scans rawget-cached children for materialization, falling through to a walking encoder only when needed. Substring fast path still fires for fully-untouched subtrees.
• Cached-proxy identity — Caught in final review: parent-level __newindex was rebuilding fresh proxies and overwriting cached child references. Fixed by snapshotting the rawget cache before nilling internals and preferring cached entries during the rebuild. Uses raw next() rather than pairs() (which would invoke __pairs and walk the FFI side).

Test plan

• cargo test --release (72 unit + integration, default features)
• cargo test --release --no-default-features (scalar scanner)
• cargo test --features test-panic --release
• make test → 75 busted Lua tests (0 failures, 0 errors)
• make bench runs cleanly; new rows produce sane numbers
• CI must reproduce all gates green

Migration story for callers

-- before
local cjson = require("cjson")
local t = cjson.decode(body)
local model = t.model
for _, m in ipairs(t.messages) do print(m.role, m.content) end
local s = cjson.encode(t)

-- after — two symbol swaps
local qd = require("quickdecode")
local t = qd.decode(body)
local model = t.model
for _, m in qd.ipairs(t.messages) do print(m.role, m.content) end
local s = qd.encode(t)

cjson.null and cjson.empty_array_mt are reused when cjson is loaded, so existing v == cjson.null checks keep working unchanged.

Related

• Spec: docs/superpowers/specs/2026-05-16-lazy-table-cjson-compat-design.md
• Plan: docs/superpowers/plans/2026-05-16-lazy-table-cjson-compat.md