jsonata: maxDepth scales with input size, and an object constructor over a $map'd lambda is quadratic

[jymaire]

jsonata: maxDepth scales with input size, and an object constructor over a $map'd lambda is quadratic

Summary

io.kestra.plugin.transform.jsonata.TransformItems and TransformValue regressed on two independent axes after #69 replaced com.ibm.jsonata4java:JSONata4Java with com.dashjoin:jsonata (shipped in Kestra 1.3.x). A flow that transformed a 1000-item batch in ~2 s on 1.2.6 now (a) fails outright with Depth=1001 max=1000 unless maxDepth is raised well above its documented meaning, and (b) takes ~2 min once it does run. Both defects are in the engine, not in user expressions.

Actual Behaviour

1. maxDepth is consumed per input item, not per recursion level

com.dashjoin:jsonata tracks the bound in Timebox: depth++ on evaluate-entry, depth-- on evaluate-exit, with both callbacks returning early when the frame carries isParallelCall. The accounting is asymmetric, so the counter never fully unwinds and grows roughly linearly with the number of items processed — even though the expression's actual nesting depth is constant.

Measured minimum workable maxDepth, expression nesting held constant ($map($, function($r) { … }) over N items):

Items	Minimum maxDepth that succeeds
25	50
100	200
500	1000
1000	~2000–3200

The trigger is not limited to lambdas: any nested object constructor evaluated once per item leaks too. Measured at N=1000 with a plain path projection, no lambda involved:

Expression shape	Minimum maxDepth
flat object, 2 keys	5
nested object inside an array literal — [{…}]	3200
array of objects + filter — [{…},{…}][pred]	3200
$lookup({…}, key) on an object literal	3200
($v := […][pred]; …) block	1600

Consequences:

• Our default of maxDepth: 1000 is effectively an undocumented cap on input size (~500 records) for any expression that nests an object constructor, which is most non-trivial ones.
• The failure message (Stack overflow error: Check for non-terminating recursive function. Consider rewriting as tail-recursive) points users at recursion they never wrote.
• maxDepth is unusable as documented ("maximum depth of recursion"), because the value a flow needs depends on how many records it happens to process that run.

2. An object constructor consuming a $map'd user-defined lambda is quadratic

The blowup needs two ingredients together: a user-defined lambda passed to $map, and an object constructor consuming its result. Either alone is fine. 45-field projection, all shapes producing the same output:

Shape at 1000 items	Time
{"items": [$.{…}]} — path projection	21 ms
{"items": $map($, $string)} — builtin function, same wrapper	10 ms
$map($, function($r){…}) — lambda, no wrapper	103 ms
$append([], $map($, function($r){…})) — lambda, array wrapper	23 ms
{"items": $map($, function($r){…})} — the reported shape	45 535 ms

Scaling of the slow shape: 1310 ms at 250 items, 7976 ms at 500, 45 535 ms at 1000 — doubling the batch multiplies time by ~5.7.

This is not the runtime-bounds machinery: with setRuntimeBounds never called the same case still takes 37 624 ms.

Since a builtin function in the identical wrapper is flat while a lambda is not, the cost appears tied to a per-invocation frame that lambda application leaves behind — consistent with defect 1, where the un-unwound counter grows by exactly one per mapped item. Object-constructor evaluation would then traverse a scope chain that grows with the array, once per key. Hypothesis, not confirmed against the library source.

Expected Behaviour

• maxDepth bounds expression nesting only. A fixed expression must need the same maxDepth whether it processes 10 records or 10 000.
• Evaluation time is linear in input size for an object constructor over a $map'd lambda, matching the path-projection and builtin-function forms.
• A depth-bound breach reports the offending construct rather than suggesting tail recursion for an expression containing none.

Reproducer

id: jsonata_maxdepth_repro
namespace: company.team

tasks:
  - id: generate_1000_items
    type: io.kestra.plugin.core.storage.Write
    extension: .json
    content: |
      [{% for i in range(1, 1000) %}{"EventId":"e{{ i }}","Value":{{ i }},"Currency":"USD"}{% if not loop.last %},{% endif %}{% endfor %}]

  - id: transform
    type: io.kestra.plugin.transform.jsonata.TransformItems
    from: "{{ outputs.generate_1000_items.uri }}"
    expression: |
      {
        "items": $append([], $map($, function($r) {
          {
            "eventId": $r.EventId,
            "depositInfos": [ { "value": $r.Value, "currency": $r.Currency } ]
          }
        }))
      }

Fails with Depth=1001 max=1000 on the default maxDepth. Setting maxDepth: 20000 makes it succeed but the task then runs for ~2 min instead of ~2 s.

The defects reproduce against the library directly, without Kestra in the loop — useful for the upstream report:

// com.dashjoin:jsonata:0.9.10
var expr = Jsonata.jsonata("$map($, function($r) { {\"x\": $r.A, \"y\": $r.B} })");
var frame = expr.createFrame();
frame.setRuntimeBounds(Long.MAX_VALUE, 1000);
expr.evaluate(listOfNMaps(1000), frame);   // JException: Depth=1001 max=1000

Logs / Stack Trace

JSonataException Stack overflow error: Check for non-terminating recursive function.
Consider rewriting as tail-recursive. Depth=1001 max=1000

Surfaced through Transform.evaluateExpression, which wraps it as
RuntimeException("Failed to evaluate expression", cause).

Environment

• Kestra version: 1.3.30 (broken) — last known good: 1.2.6
• Plugin version: plugin-transform as bundled with the versions above; JSONata engine com.dashjoin:jsonata:0.9.10
• Deployment: not specified by the reporter; reproducible standalone on Linux / OpenJDK

Additional Context

• Introduced by Replace JSONata4Java with dashjoin/jsonata-java #69 (Replace JSONata4Java with dashjoin/jsonata-java for full JSONata compatibility), which resolved consider changing JSONata Java implementation to overcome feature limitations of JSONata4Java #40. The compatibility win is real — this issue is about the performance and bounds-semantics regressions that came with it.
• com.dashjoin:jsonata:0.9.10 is the latest release on Maven Central, so there is no version to bump to. Both defects need upstream issues filed at dashjoin/jsonata-java; defect 1 has a clear candidate cause in Timebox's entry/exit isParallelCall asymmetry.
• Related prior work on the same code path: fix(jsonata): lower default maxDepth to prevent Windows worker crash #79 and fix(jsonata): prevent StackOverflowError on small JVM stacks via eval thread isolation + lower default maxDepth #80 (StackOverflowError on small worker stacks, 4 MB eval-thread isolation, maxDepth default churn). fix(jsonata): prevent StackOverflowError on small JVM stacks via eval thread isolation + lower default maxDepth #80's final commit restored maxDepth: 1000 to avoid a breaking change; that default turns out to be sound once the accounting is correct — real expressions of this complexity peak around 11 nested levels.

Mitigation available today

Rewriting $map($, function($r){ … $r.X … }) as a path projection $.{ … X … } (dropping the $r. prefixes) produces byte-identical output and removes the quadratic cost. Measured on the reporter's full expression at their batch size of 1000 items, with defect 1 fixed:

Their expression	Time	Peak nesting depth
as written	17 979 ms	11
rewritten as a path projection	18 ms	11

Worth noting the depth column: the expression genuinely nests 11 levels. The maxDepth: 20000 the reporter had to set to get it running is an artefact of defect 1 alone — once that is fixed, the default of 1000 covers this expression at any batch size.

Status

Defect 1 is fixed in #103, entirely within this repository — the callback setters setEvaluateEntryCallback / setEvaluateExitCallback are public, so the accounting can be done correctly without waiting on upstream.

Defect 2 cannot be fixed here; it needs an upstream change. Users are not blocked in the meantime, since the path-projection rewrite above is linear. Only expressions that genuinely need a function value — a lambda passed to $reduce or $sort, or keys built conditionally per item — have no rewrite available.

• Fix the depth accounting so maxDepth is independent of input size (fix(jsonata): bound maxDepth by expression nesting, not input size #103)
• Correct the maxDepth @Schema wording, which described a recursion budget the counter never measured (fix(jsonata): bound maxDepth by expression nesting, not input size #103)
• Document the path-projection preference in src/main/resources/doc/io.kestra.plugin.transform.jsonata.md (fix(jsonata): bound maxDepth by expression nesting, not input size #103)
• Add regression tests asserting a fixed expression needs a fixed maxDepth across input sizes (fix(jsonata): bound maxDepth by expression nesting, not input size #103)
• File both defects upstream at dashjoin/jsonata-java with the standalone reproducers
• Once defect 2 is fixed upstream, drop the doc caveat; once defect 1 is fixed upstream, consider reverting to setRuntimeBounds

---

View as Artifact