Add qjson.next helper for lazy proxy migration

[membphis]

Spec: qjson.next helper for lazy proxy migration

Background

qjson.decode returns lazy object/array proxies instead of plain Lua tables. The project already provides qjson.pairs, qjson.ipairs, and qjson.len so code can avoid relying on LuaJIT builds with LUAJIT_ENABLE_LUA52COMPAT.

A remaining migration gap is call sites that use manual next loops instead of pairs, for example:

local k, v = next(t)
while k ~= nil do
    -- process k/v
    k, v = next(t, k)
end

Lua/LuaJIT does not provide a __next metamethod, so native next(t, k) cannot be made to traverse qjson lazy proxies. Without a parallel helper, these call sites cannot be mechanically migrated while preserving lazy decoding.

Goals

Add an explicit qjson.next(t, prev_key) helper that provides a low-friction migration target:

local k, v = qjson.next(t, prev_key)

It should also work with Lua's generic-for protocol:

for k, v in qjson.next, t, nil do
    -- process k/v
end

Dispatch

qjson.next selects behavior by metatable identity, in three cases:

• LazyObject → ordered-snapshot iteration (see below).
• LazyArray → numeric index iteration 1, 2, ..., n.
• Anything else (plain tables, empty_array_mt tables, and values produced by qjson.materialize) → delegate to native next(t, prev_key).

Note the asymmetry between the two proxy kinds after mutation:

• A mutated LazyObject keeps its LazyObject metatable, so it stays on the ordered-snapshot path.
• A mutated LazyArray is converted in-place to a plain table tagged with empty_array_mt and loses its LazyArray metatable. It therefore falls into the native-next branch. As a result, the ascending-order guarantee below applies only to unmutated lazy arrays; once an array is mutated it follows native next ordering (unspecified) like any plain sequence. This is acceptable and intentional.

API semantics

• For plain Lua tables, qjson.next(t, prev_key) delegates to native next(t, prev_key).
• For (unmutated) lazy arrays, it iterates numeric indexes in ascending order: 1, 2, ..., n.
• For lazy objects, it provides Lua-table-style key semantics:
  • each key is returned at most once;
  • duplicate JSON object keys use last-wins values;
  • key order follows first appearance in the source JSON. (This is a stronger guarantee than native next, whose order is unspecified; callers migrating from native next must not have depended on order, but where order matters this helper is deterministic.)
• For already materialized/mutated lazy objects, it reflects the current ordered object state, including added, updated, and deleted keys (see "Snapshot lifecycle & mutation visibility").
• For lazy objects whose child proxies were mutated before the parent is iterated, returned values preserve those cached child mutations (same proxy identity as t[k]).

Example duplicate-key behavior:

local t = qjson.decode('{"a":1,"a":2}')
local k, v = qjson.next(t)
-- expected: k == "a", v == 2

Stale / missing prev_key

If prev_key is not present in the iteration's effective key set (e.g. it was deleted, or never existed), qjson.next must behave like native next does for an invalid key: raise an error ("invalid key to 'qjson.next'"). It must not silently return nil, which would mask iteration bugs. Passing nil always (re)starts iteration from the first entry.

Terminal / empty cases

• qjson.next(t, nil) on an empty object or empty array returns nil.
• Iterating past the last entry returns nil (loop termination).

Snapshot lifecycle & mutation visibility

The lazy-object key snapshot is built lazily on the first qjson.next(obj, nil) call for that object and then cached on the proxy. Two consequences must be specified, because "cached snapshot" and "mutations are visible" can otherwise conflict:

1. Mutation before iteration is visible; mutation during iteration follows native next rules. A mutation applied before the first qjson.next(obj, nil) of an iteration is reflected. Once iteration has started (snapshot built), qjson.next matches the native next contract: assigning to an existing field may or may not be reflected by the in-flight traversal, and adding a new field is undefined. Callers must not add keys mid-traversal.
2. ORDER_* takes precedence over the private snapshot. When the object already carries materialized ORDER_KEYS/ORDER_VALUES state (because it was mutated), qjson.next reads that live state directly so adds/updates/deletes are reflected. qjson.next must never create ORDER_* itself — only read it when already present. Only when ORDER_* is absent does it build and use a private snapshot (see Implementation notes).

The cached snapshot is retained on the proxy for the proxy's lifetime (until GC). This is an O(n) memory cost in the object's entry count, accepted as the price of table-style traversal; it is not freed when a single iteration completes.

Non-goals

• Do not monkey-patch global next.
• Do not attempt to change LuaJIT or introduce a __next runtime extension.
• Do not make qjson.next the precise JSON object iterator. qjson.pairs should remain the API that can preserve source-order traversal and duplicate-key visibility.
• Do not require callers to materialize the whole decoded value into a plain Lua table.

Implementation notes

qjson.next must avoid reusing the existing ORDER_KEYS / ORDER_VALUES state for unmodified lazy objects, because the presence of ORDER_KEYS is the flag that flips qjson.pairs from JSON-entry semantics (source order, duplicate-visible) into Lua-table semantics (deduped, last-wins). Building ORDER_* as a side effect of qjson.next would silently change later qjson.pairs traversal. A separate internal snapshot/cache (e.g. private sentinel keys NEXT_KEYS / NEXT_VALUES) is required for unmodified objects.

Concretely:

• Precedence. If ORDER_KEYS exists, iterate it (mutated object — qjson.pairs is already in Lua-table mode, nothing to protect). Otherwise build and cache the private snapshot. Never create ORDER_* from qjson.next.
• O(n) traversal. Maintain a key→ordinal index (NEXT_INDEX) alongside the ordered key list so qjson.next(t, prev_key) resolves the successor in O(1); a linear scan per call would make a full traversal O(n^2).
• Container values come from CHILD_CACHE. When building the snapshot, container values must be sourced from the existing CHILD_CACHE (as the pairs stateful iterator already does) so that pre-iteration child-proxy mutations are visible and child proxy identity matches t[k]. A parent dirtied only via a child mutation has _dirty = true but no ORDER_*; it still takes the private-snapshot path and must read CHILD_CACHE.

The snapshot scan cost is acceptable because the helper is explicitly used for table-style traversal.

Acceptance criteria

• qjson.next is exported from require("qjson").
• qjson.next works for plain Lua tables by delegating to native next.
• qjson.next iterates unmutated lazy arrays in ascending numeric order.
• A mutated lazy array iterates via the native-next fallback (no crash; ordering unspecified).
• qjson.next iterates lazy objects with first-appearance order and last-wins duplicate-key semantics.
• Calling qjson.next on a lazy object does not change qjson.pairs behavior for duplicate keys.
• Mutations applied before iteration (add / update / delete, including via ORDER_*) are visible through qjson.next.
• A child proxy mutated before parent iteration is returned by the parent's qjson.next as the same proxy instance, reflecting the mutation.
• qjson.next(t, nil) returns nil for empty objects/arrays; iterating past the end returns nil.
• qjson.next raises an error for a prev_key not in the effective key set (matching native next).
• qjson.next does not create ORDER_KEYS/ORDER_VALUES on an otherwise-unmodified object.
• Documentation explains that qjson.next is a migration helper for manual next loops and that native next itself cannot be customized for lazy proxies.

Test plan

Add Lua busted coverage for:

• plain table fallback;
• lazy array iteration (ascending order);
• mutated lazy array iteration via native-next fallback;
• lazy object iteration (first-appearance order);
• duplicate object keys with last-wins behavior;
• qjson.next followed by qjson.pairs still seeing duplicate entries (i.e. qjson.next did not materialize ORDER_*);
• object mutation (add/update/delete) before iteration is visible;
• cached/mutated child proxy visibility and identity through parent qjson.next;
• empty object and empty array return nil;
• iterating to the end returns nil;
• stale/missing prev_key raises an error.

Update README and docs/migrating-from-cjson.md with the migration pattern:

-- before
local k, v = next(t, k)

-- after
local k, v = qjson.next(t, k)

[coderabbitai]

🔗 Related PRs

#60 - feat: preserve key order in decode-modify-encode workflow [merged]
#93 - perf: add stateful object iterator [merged]
#105 - test: add lazy mutation property coverage [merged]
#108 - feat: align Lua encode with cjson [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[membphis]

Closing as not planned for now.

After fleshing out the spec, the cost/benefit does not hold up on its own:

• The common idioms behind manual next loops are already covered by existing helpers:
  • emptiness check next(t) == nil → qjson.len(t) == 0
  • "grab any one entry" local k, v = next(t) → for k, v in qjson.pairs(t) do return k, v end
• The only use that qjson.pairs genuinely cannot replace is resumable, position-by-key incremental iteration across calls — a rare pattern in cjson-consuming code.
• The chosen (t, prev_key) API shape forces a lot of machinery for little payoff: a third iteration semantics (next dedups, pairs keeps duplicates — opposite behavior in the same library), a retained O(n) key/value snapshot that cuts against qjson's parse-once / extract-a-few / discard design, plus stale-key errors, snapshot-lifecycle rules, ORDER_*/NEXT_* precedence, and CHILD_CACHE sourcing.

If a concrete migration need surfaces — i.e. real call sites doing resumable next(t, k) loops over decoded JSON that cannot be mechanically rewritten to qjson.pairs / qjson.len — we can reopen with that evidence; that would immediately justify the helper.

In the meantime, the cheaper path is to document the pairs / len migration patterns above in docs/migrating-from-cjson.md, with no new API, no second iteration semantics, and no retained snapshot.