Release Release v2.2.12 - "Gymkata" · txjmb/jsonata-core

Release v2.2.12

Changed

The mimalloc allocator no longer falls into its aligned slow path. With the mimalloc
feature (the published wheels), every allocation went through mi_malloc_aligned. There,
once the free list of a size class whose block size is not a power of two runs dry, each
allocation of that size over-allocates from a larger class instead of refilling it (about
110 extra instructions per allocation), and which sizes are hit depends on what else
shares their class. The global allocator now calls mi_malloc for the alignments mimalloc
guarantees (up to 16 bytes on 64-bit targets, 8 on 32-bit, for sizes that are a multiple
of the alignment), which is nearly every Rust allocation, and uses the aligned API for the
rest. Callgrind on the A/B benchmark expressions: 0.2% to 3.6% fewer instructions on every
one, both engines (Sort 3.6%, Deep Path 2.8%, Parent 1.9%).
An object constructor or .(expr) path step is compiled once per evaluation, not each
time it runs (#197). The evaluator
keeps each step's compiled form, so a step inside a function body, such as $map(big, function($v){ $v.items.{"a": x, "b": y} }), compiles once instead of once per call (about
10% fewer instructions for 200 calls), and a step that does not compile is not retried.
Whether a builtin the step calls is shadowed is still checked every time it runs. Rust API:
PathStep has a new id: StepId field, the key for that cache (StepId::default() in a
struct literal, which gives a new id; PathStep::new and with_stages set it). It is
ignored by equality and serde, and code that rewrites a step's node should give it a new
id. AstNode stays Send + Sync.
A $ step over an array known to hold no arrays skips the nested-array scan
(#196). JArray gains a private flat
flag, read with is_flat() and set with JArray::new_flat or JValue::sequence_flat, for
arrays known to hold no array a path step flattens. Input conversion (JSON, serde_json,
Python lists), numeric ranges, path steps that flattened nothing, field lookups over arrays
and [...] constructors over flat parts set it where knowing costs nothing; make_mut and
with_items clear it. $sum(values.$) over 1,000 numbers runs about 14% fewer instructions
(callgrind), and results are unchanged, including n.$ over nested arrays.
A filter, sort or group-by right after a sort of a tuple stream sees each tuple as an
object (#214). jsonata-js's sort returns
a tuple stream's tuples as a plain array, so the stage that follows reads the tuple, not its
value: acct.Order.P^(>%.id)[N = "C"] is undefined (it was {"N":"C"}), a second sort such
as ^(N) keeps the order, {N: N} gives {}, and o.t^(>%.id)^($) is T2008. This holds
after a position filter such as [0] too. Later path steps see the values as before, and a
sort used as a value ($x := acct.Order.P^(>%.id), $string(...)) gives the values rather
than the internal tuples.
A group-by on a path groups the whole path
(#208). In jsonata-js the {...} is
the path's group, and later . steps, calls, filters, sorts and binds extend that same path
before it is grouped. So acct.Order.P{N: N}.A groups acct.Order.P.A and gives {} (it
was "A"), acct.Order.P{N: N}[0] filters step P (it was S0209), and a second group-by
on the same path, as in acct.Order{id: id}.P{N: N}, is S0210. A group-by on a block, $x
or * still applies to that step alone, and [] after it keeps the array instead of
raising S0209.
Rust API: JValue::Array holds a JArray instead of an Rc<Vec<JValue>> (#184). JArray
dereferences to Vec<JValue>, so code that reads arrays is unaffected. Code that builds them
calls JValue::array(vec), and JArray::make_mut, into_vec and ptr_eq replace
Rc::make_mut, Rc::try_unwrap and Rc::ptr_eq. JArray also carries the sequence, cons
and keep_singleton flags jsonata-js keeps on arrays, and an outer_wrapper flag for the
top-level input array.
Results are unwrapped and flattened by jsonata-js's two rules (#184). A path step's
results are flattened one level, except for arrays built by [...], and on the last step a
single array result is returned as is. After every expression, an empty sequence is
undefined and a one-item sequence is its item unless [] applies. These replace the
engine's own heuristics, so results now match jsonata-js 2.2.2 in cases such as
[{'a':[[1]]}].(a) ([[1]]), n.$ over nested arrays, $lookup and * (which flatten
one level), x[]@$v (now accepted) and a range that counts down ([3..1], undefined).
A top-level array input is one item to a path, as jsonata-js's outerWrapper makes it,
while $, * and an object constructor see the bare array: {"x": a[0]} over
[{"a":[1,2]},{"a":[3]}] is {"x":[1,3]}. $eval(expr, array) wraps its focus the same way.
An object constructor groups an array input's items by key, as jsonata-js
evaluateGroupExpression does: a value is evaluated over its group's items, so
[[1,2],[3]].{"x": $} is {"x":[1,2,3]} and over [4] it is {"x":4}. A group-by's keys
keep the order they are first produced in.
function and λ are names, not keywords. They open a lambda only when immediately
followed by (, matching jsonata-js, which has no such keyword and decides lambda-ness from
the callee of a call. Anywhere else they are ordinary path steps, so function now reads a
field called function (and λ a field called λ) instead of being a parse error.
A / after a bare function/λ divides instead of opening a regex literal. The lexer
treated the lambda opener as an operator for this purpose, but a lambda's next token is always
( — so a / can only follow one of these when it is a name, and a name is a value.
function / 2 is now 21, as upstream.
A lambda's signature is validated when it is parsed, not when it is first called — but
only for the two errors jsonata-js also raises at parse time, S0401 and S0402. Every other
signature complaint still surfaces at call time as before.

Fixed

Recursion through a tail-called partial application no longer crashes the process
(#255).
($f := function($n){ $n <= 0 ? 1 : $f($n-1, ?)() }; $f(2000)) is 1, as in jsonata-js; it
overflowed the native stack at about 2,000 levels on both engines and killed the Python
process. A call of a value in tail position that is not a function with tail calls (a
partial application, also of a builtin such as $reduce(?, $g, 0), or a builtin called as
a value, ($reduce)(xs, $g, 0)) was applied by the trampoline outside any evaluation
step, so a recursion through it bypassed both the native-stack guard and the depth limit.
It now runs under both: deep recursion evaluates, and with no base case it raises U1001 at
the 30,000-level limit.
A tail-recursive loop no longer stops at 100,000 iterations
(#251). The tail-call trampoline had a
hardcoded 100,000-iteration cap when no timeout was set, so
($loop := function($n){ $n = 0 ? "done" : $loop($n - 1) }; $loop(200000)) raised
"U1001: Stack overflow - maximum recursion depth (500) exceeded" on both engines, naming a
depth nothing enforced. jsonata-js has no such cap and returns "done". Tail calls now run
without any limit: a tail loop keeps one frame, so a million iterations take about 0.4s
with no memory growth. max_stack_depth (D1011) and the 30,000-level U1001 backstop still
bound non-tail recursion, and a timeout (D1012) is what stops a runaway tail loop. The
U1001 message now names its real limit and follows the wording of the jsonata-js test
runner's U1001: "Stack overflow error: evaluation nested deeper than 30000 levels. Check for
non-terminating recursive function. Consider rewriting as tail-recursive."
$eval with an undefined focus evaluates against the current input, as in jsonata-js
(#247). $eval("a", $v) with $v
unbound used the undefined focus as its input and was undefined; it now reads a from
the input, as $eval("a") does. A defined focus still replaces the input.
A builtin called in a function's tail position returns its result as jsonata-js does
(#250). jsonata-js runs such a call
from its trampoline, which never unwraps the result, but it was unwrapped here, so an
empty sequence became undefined: $map([1,2], function($v){ $keys({}) }) was undefined
instead of [[], []], and $keys({"a":1}) there gave "a" instead of ["a"]. A direct
call such as $f() still unwraps it. With this, $keys of undefined and $lookup over
an array that matches nothing give an empty sequence, as in jsonata-js; a conditional
without an else in that position is undefined, not null; and $sort tests its
comparator's result with JavaScript truthiness, so a comparator returning [] or {}
swaps. $split of undefined is undefined, not null, so
$map([1,2], function($v){ $split(nothing, " ") }) is undefined instead of
[null, null] and [$split(nothing, " ")] is []; $sift(nothing, $f) is undefined
instead of an error.
A builtin called in a function's tail position runs with the input jsonata-js gives it
(#247,
#250). jsonata-js's trampoline applies
it with the input of the function's own application: null when the function is a
callback of a builtin such as $map or is applied by ~> to a non-call, and the call
site's input for a direct call. It used the function body's input instead, so
$map(["a"], function($e){ $string() }) gave the input instead of "null",
$map(["a"], function($e){ $eval("a") }) gave 5 instead of undefined, and
($f := function(){ $eval("a") }; {"a":7}.$f()) read the outer input instead of
{"a":7}. This affects only what such a builtin reads from the context: $eval's
focus, and a signature's - parameter left without an argument. A builtin that is
itself a function's tail call applies the functions it is given with the input they were
defined with, as jsonata-js's trampoline passes them unwrapped:
($o := function(){ $map([1], function(){ $eval("a") }) }; $o()) is 5. Known gap
(#254): jsonata-js wraps a function
passed as an argument of a direct call in a closure that applies it with a null input,
which is not modelled yet, so
($h := function($f){ $f("x") }; $h(function($e){ $eval("a", $nope) })) gives 5 instead
of undefined.
A string literal heading a path reads that field under its predicates, group-by, sort or
bindings. "s"{k: v}.p, "s"[0].k, "s"#$i.k and "s"^(v).k read field s, as in
jsonata-js, whose . makes every string step of a path a name; only a bare "s".k did, so
these read the string itself and gave undefined. "s"{k: v}.% is the input (it was S0217).
~> into a call of an expression's value passes the left side as the first argument.
"x" ~> $uppercase[0]() is "X" and 1 ~> $o[1]() calls $o[1] with 1, as in
jsonata-js; only a call of a named function got the left side, so these called the function
without it (T0410) or applied its result (T2006).
A called path naming a function without its $ raises T1005, as in jsonata-js:
uppercase#$i(), uppercase[0]() and ($x := 1; x#$i()) suggest $uppercase and $x
instead of raising T1006. As upstream, the hint is given when the path gives nothing and
its first step names a builtin or a binding with a JavaScript-truthy value, and not when
that step is a quoted string after . (a."uppercase"[0]() is T1006).
Calling a quoted string step after . calls the string, raising T1006. In jsonata-js,
( takes "f" (or "f"[0], "f"#$i) as its procedure before . makes the path's string
steps names, so $o."f"("a") and arr."uppercase"() are T1006; they called the function in
field f, or raised T1005.
Calling a step with an index or focus binding calls what the step gives.
($o := [function($x){$x}]; $o#$i(3)) and $o.g[0]#$i(3) are 3, as in jsonata-js, where
# and @ bind tighter than . and ( takes the bound step as its procedure; they were
parse errors (T1006). A bound last step of a longer path is called in each context item.
$o[0]@$v(3) now raises the S0215 that $o[0]@$v does, instead of T1006.
A filtered array constructor inside an array constructor keeps its nesting
(#249), as in jsonata-js, which pushes
a member that is an array constructor whole, with any predicates or [] after it.
[[0..9][$ % 2 = 0]] is [[0,2,4,6,8]] and [[1, 2, [3, 4]][-1]] is [[3,4]] (they were
[0,2,4,6,8] and [3,4]), and so for [[1,2][]], [[[1,2]][0]], [1,[2,3][$>0]] and
a constructor in an object value; $count([[1,2][$>0]]) is 1 (it was 2). A member that is
a path is still spliced in: [([1,2])[$>0]], [[1,2]#$i] and [[3,1,2]^($)] are flat.
A filter on a step that gives nothing is still evaluated once, as in jsonata-js, whose
evaluateFilter filters [undefined] there: x[$string(1,2,3)] raises T0410 and
x[[a()]] T1006 when x is missing (they gave undefined). The result is still undefined,
and a filter that selects nothing (x[false], x[0]) stops the later ones. As there, a
sort step's filters do not run when the sort gives nothing (x^(a)[$error("e")]), nor do
those on the call after ~>. A filter that calls nothing (x[$ > 5]) is not run on an
item without the field, as it cannot raise an error there.
Calling a bound literal is the same T1006 as calling the literal. "f"@$v(),
a.1@$v() and a.true#$i() raise T1006 while parsing, as "f"() and a.1() do, ahead of
a literal-step S0213 or another member's error ([$string(1,2,3), "f"@$v()]). A bound
.(expr) step is called as its block: a.(1)@$v() is T1006 (it had no error code). A @
on a sort or group-by under a unary minus after . comes after that .'s S0213
(true.-a^(x)@$v is S0213), as both bind looser than . in jsonata-js.
A misplaced @ binding is reported before a literal path step, as in jsonata-js,
whose processAST checks a @ step before the . around it checks its steps:
1[0]@$v.**.q, null[0]@$v.$append(a.c.$, n[]) and 1.a[0]@$v are S0215 and
1^(x)@$v.a is S0216 (they were S0213 since the S0213 check below covered every step).
A literal used as a path step is S0213 wherever it is in the path
(#248), as in jsonata-js, which checks
every step of a path built by .. Only a literal right of a dot was rejected, so null.$,
true.$, 2#$q.a, null@$x.a, 2[0].a, null^(x).a and null.(1) were evaluated, and
a.-2 gave [-2,-2] (jsonata-js folds -2 into a number first). A number followed by .
and no digit is now the number and a path step, as in jsonata-js's tokenizer, so 2.a and
1.5.a are S0213 too (2.a was "Invalid number"), and an exponent with no digits (2e)
is S0201. The same check turns a string step into a field name anywhere in the path:
"a"[0].b, "a".(b), "a".[b] and "a"#$i.b read the field a (they used the string).
Creating a function with a deeply nested body no longer crashes deep in a recursion.
Creating a function clones its body and collects its free variables, two recursive walks
over the body that ran without the stack guard deep evaluation uses. A body nested a few
hundred levels deep, created inside a recursion thousands of calls deep, could overrun
the stack headroom left there and crash the process (a native stack overflow, seen on
Windows with a 300-level body). Both walks now move to a new stack segment when the
stack runs low, as the value walkers do.
A null sort term raises T2008 whatever the term's form, as in jsonata-js. A term that is
not a plain field name (a.b, $.a, a function call) turned a null result into undefined,
so [{"a":{"b":2}},{"a":{"b":null}}]^(a.b) and [{"a":2},{"a":null}]^($.a) returned the
input unsorted; a plain field such as ^(a) already raised T2008. The null is now kept, and
jsonata-js's comparator raises T2008 when it compares it with another defined term. A null
beside only undefined terms, or in a one-item sort, is never compared, so it does not raise
and sorts before the undefined ones: [{},{"a":null}]^($.a) is [{"a":null},{}] (it was
[{},{"a":null}]).
Calling an indexed variable calls the function it gives. ($o := [function(){1}]; $o[0]()) is 1 and $string[0](1) is "1", as in jsonata-js, which evaluates a call's
procedure expression and calls the result; they were parse errors ("Invalid function name"),
and T1006 in v2.2.11, so $exists($o) ? $o[0]() : "none" failed to compile instead of giving
"none". An indexed field as a path's last step is called in each context item, as ( binds
tighter than . in jsonata-js: $o.g[0]() gives what g[0] returns (it was T1006). An
index that gives no function is still T1006 (T1008 for a partial application).
A group-by on a leading array constructor followed by another step is undefined again,
as in jsonata-js and v2.2.11. jsonata-js groups on the constructor node and evaluates that
step once, so the next step reads the grouped object as an input with no items. Since the
path-group change for #208 the step ran
once per item instead: [1,2]{"k": $}.k gave [1,2] and [1,2]{$string($): $}.[nothing]
gave []. These, [1,2]{"k": $}.{"z": $}, [1,2][]{"k": $}.k and a group on a block
starting with a constructor (([1,2].$){"k": $}.k) are undefined. [1,2]{"k":$} is still
{"k":[1,2]}, ([1,2]){"k":$}.[5] is still [5] and [1,2]{"k":$}[0] is still S0209.
Freeing a very deeply nested value no longer overflows the stack
(#233). A value was dropped by plain
recursion, one stack frame per level, so an array or object nested more than about 10,000
deep (easy to build with $reduce, e.g. $reduce([1..100000], function($a, $i){ {'a': $a} }, {})) crashed the process when it was freed. So did reading down a deep Python input
($reduce([1..99999], function($x, $i){ $x.a }, $) over {"a": {"a": ...}}), as each lazy
dict wrapper holds the next. The payloads behind an array's and an object's Rc, and lazy
dict wrappers, now check the stack before freeing values that hold containers, and once it
runs low they take the rest of the value apart on a heap work-list instead of recursing. A
small payload that frees no container (the common leaf object) skips the check after a
look at its values; a lazy Python dict wrapper records whether it holds a container, and
a chain of wrappers read down from one another checks once every 32 levels. Only what
this drop frees is taken apart; a shared value is just released. Callgrind on the A/B
benchmark expressions over Python input, against main with the allocator change above
(instructions per evaluation from the difference between 600 and 100 evaluations): at
most 0.72% more (the 5-level path lookup on the tree-walker, under 100 instructions),
0.5% to 0.65% more on the %, #, @ and top-rated expressions, and fewer on about half.
Rust API: JValue::Object holds Rc<JMap> instead of Rc<IndexMap<String, JValue>>, and
JArray's items sit in a JItems. JMap dereferences to the IndexMap, and JItems to
Vec<JValue>; both iterate by reference, and JMap collects from (String, JValue) pairs.
JValue::object still takes an IndexMap. What breaks: JValue::Object(Rc::new(map))
needs Rc::new(map.into()) or JValue::object(map); Rc::try_unwrap, Rc::make_mut and
Rc::unwrap_or_clone on an object's payload yield a JMap (use .into_inner(), or use it
through deref); LazyPyDict::to_object() returns Rc<JMap>. Not covered: closures. A
chain of tens of thousands of closures, each capturing the previous one, is still freed by
recursion.
A function body resolves names where the function is defined, not where it is called
(#191). A binding in the caller's scope
leaked into the body: ($f := function($x){ $string($x) }; ($string := function($x){"z"}; $f(1))) is "1" as in jsonata-js (it was "z"), and ($g := function(){ $v }; $f := function($v){ $g() }; $f(1)) is undefined (it was 1). The same holds for transforms,
partial applications, tail calls, and internal temporaries such as $__temp_arg_0, and to a
closure returned from another function, which resolves through its maker's defining scope and
never its maker's callers. While the defining scope is still running, a binding made there
after the function is seen too; once that scope has finished, only the bindings the function
captured when it was defined remain (jsonata-js keeps the whole scope alive, so
($f := ($g := function(){ $y }; $y := 5; $g); $f()) is 5 there and undefined here). A partial
application such as $uppercase(?) now resolves its function when it is made, as jsonata-js
does: rebinding $uppercase afterwards no longer changes it, and partially applying an
unbound name is T1008 when the partial is made. Every compiled fast
path now checks builtin shadowing through one rule, so $map/$filter/$reduce callbacks,
object constructors, predicates and fused aggregates call a builtin only where the
tree-walker would; a parameter named like a builtin (function($string){ $string(1) }) now
shadows it there too. Calling a name bound to something that is not a function, such as
$string := 5; $string(1), is T1006 (T1008 when partially applied) instead of calling the
builtin. Deep recursion no longer slows down with depth when its body calls a builtin, since a
lookup skips the caller frames. $eval runs in its caller's scope without seeing the
evaluator's temporaries, and a binding it makes lands there, even when it is reached through
$map: ($m := $map; $m(["$w := 3"], $eval); $w) is 3. A composition f ~> g calls g in
the scope it is called from and f in a call frame of its own below the scope where ~> was,
as jsonata-js's chainAST does, so ($x := 7; ($string ~> $eval)("$x")) is 7 (it was
undefined) and ($c := $eval ~> $string; $c("$z := 5"); $z) stays undefined. g's call is a
tail call, so ($inc ~> $loop)($n) in tail position loops without growing the stack. $eval
as a lambda body's tail call runs where the chain of tail calls began, as jsonata-js applies
a tail call's thunk: ($v := "root"; $h := function(){ $eval("$v") }; ($v := "caller"; $h())) is "caller", while ($r := $eval("$v"); $r) in the body still reads the body's own
scope. A tail-call loop reuses its frame for the next pass unless a function that can see
the frame outlives the pass; functions the pass makes and drops, including ones kept in
local arrays, objects or other functions, or tail-called, do not count, so such loops run in
constant memory. A function that does live on, such as one passed to the next pass, keeps
its frame, so a loop that passes a new closure on each pass still grows with the number of
passes. That is a cost of lexical scoping, as in jsonata-js: a loop that collects a closure
on each pass into an accumulator ($f($k - 1, $append($acc, $c))) keeps every pass's frame
with all its bindings, the accumulator so far included, so its memory grows with the square
of the number of passes (about 1.1 GB at 5,000 passes, where it was 41 MB). A builtin such as
$map run as a tail call applies its function arguments where the chain of tail calls began,
as jsonata-js does ($sort excepted, since jsonata-js fails that call with "comp is not a
function"); recursion through such a callback, or through a tail-called composition, goes as
deep as other recursion and is U1001 without a base case, never a native stack overflow.
Still different from jsonata-js: the right side of ?: and ?? is not a
tail position here; a partial application of a user function does not carry the chain of
tail calls, and a $eval(?) partial runs where it is called, not where it was made; function
arguments of a builtin called outside a tail position are applied where the builtin runs, not
in jsonata-js's wrapper over the caller's environment; a tail $eval takes the call
site's context as its input; and a composition's first function flattens one more level of
nested one-item sequences than jsonata-js when its result comes from such a tail-called
builtin (($r ~> $string)(4) with a $r nesting $map is "[[0]]", "[[[0]]]" there).
A sort over one item gives that item, not a one-item array
(#221). jsonata-js sorts as a step after
its input, which flattens the input into a sequence, and returns a sequence of zero or one
item as is, so it is unwrapped: acct.Order[1].P^(N) is {"N":"C"} (it was [{"N":"C"}]),
[{"N":"C"}]^(N) likewise, and a sort over an empty array is undefined (it was []). A []
on the sorted path keeps the array (P[]^(N), P^(N)[], $#$i[]^($)); a [] on a block,
variable or constructor before the sort ($x[]^(p), $x[]#$i^(p)) does not. A sort whose
input is not a path (a block, literal, function call or array constructor) evaluates it once
per item of an array input, as a path's first step: over [{"a":[2,1]},{"a":[3]}],
(a[0])^($) is [2,3] (it was 2) and $count(a[0])^($) is 2 (it was 1). A sort of
objects that only look like internal tuples ({"@": 2, "__tuple__": true}) now orders them as data instead of leaving
them unsorted. A group-by after a sort groups a single array as one item (z^($){"k": $} is
{"k": [5]} for z: [[5]]), an object constructor inside a .( … ) block reads $ as its
own context ([1].([[5]].{"v": $}) is {"v": 5}), and a field sort term over an array item
reads its items' fields, so [[{"p":2},{"p":1}],[{"p":3}]]^(p) raises T2008 as in jsonata-js.
The order-by clause sorts with jsonata-js's merge sort and comparator: a missing term sorts
last in either direction ([{"p":1},{"q":1}]^(>p) keeps its order; it put {"q":1} first),
and is checked before the types, so [{"p":null},{"q":1}]^(p) keeps its order instead of
raising T2008. A leading array constructor with a focus or index binding is evaluated once,
the binding ignored, when a . follows it (jsonata-js consarray), so [$]@$w.($) over
[1,2,3] is [1,2,3] and [a]@$w.{"v": $} over [{"a":1},{"a":2}] is
{"v": [{"a":1},{"a":2}]}.
A higher-order function evaluates its callback once, to a function value
(#204). $map, $filter, $reduce,
$single, $sift, $each and $sort evaluate a callback that is not an inline lambda or a
variable once, then call the function it gives with as many arguments as it takes, as
jsonata-js's hofFuncArgs does. Before, the expression was evaluated again for each item
with the item as context, so $map([1,2], ($string)) and $map([1,2], [$string][0]) gave
functions instead of ["1","2"], $filter([1,2], (function($v){$v>1})) was undefined
instead of 2, and a host function inside the callback ran once per item. $reduce raises
D3050 for any callback taking fewer than two arguments, so $reduce([1,2], $sum ~> $string)
and $reduce([1,2], $string) are D3050. The nine higher-order builtins have their
jsonata-js arities, so $map([[3,1]], $sort) hands $sort two arguments.
A regex literal is a function value
(#205). As in jsonata-js, /re/ called
with a string gives its first match as {match, start, end, groups, next}, with start and
end in UTF-16 code units, groups always present and next() giving the following match
(undefined after the last, D1004 for a zero-length one). It can be bound
(($f := /a/; $f("xa"))), passed as a callback ($filter(["ab","cd"], /b/)), composed on
either side of ~> ($sort ~> /a/), and passed to $match, $replace, $split and
$contains from a variable. $type(/a/) is "function", $string(/a/) is "",
$boolean(/a/) is false, and a regex returned to Python is None. "xa" ~> /a/ goes
through the general ~> path, so its result has groups and next. A regex now reads as
one after :=, (, :, =, ? and the other operators that were read as division
before ($string(/a/) was S0211), m.next() calls a field holding a function, and ~>
binds at 40 as in jsonata-js, so "é" & "a" ~> /a/ matches "éa". Deliberate
divergences: an undefined subject gives undefined (jsonata-js matches the text
"undefined"), and each match's next() continues from where that match ended, while
jsonata-js's continues from wherever the regex last stopped, shared by every match it
gave: ($m := /a/("xaya"); [$m.next().start, $m.next().start]) is [3, 3] here and [3]
in jsonata-js.
Non-tail recursion goes about 10,000 calls deep
(#200). Evaluation stopped at 302 levels
with U1001, so ($f := function($n){ $n <= 0 ? 0 : 1 + $f($n - 1) }; $f(150)) failed where
jsonata-js gives 150. jsonata-js has no fixed limit; here U1001 now comes at 30,000 levels,
about 10,000 calls, on both engines. Deep levels run on stack segments allocated as needed,
so this holds on a 1MB thread stack (Windows' default) too. Converting a deeply nested result
to and from Python, serializing it, $string, = and ** likewise no longer overflow a
small stack. max_stack_depth still raises D1011. A compiled sub-expression nested more than
128 levels deep now runs on the tree-walker. A Python host function called deep inside a
recursion gets a stack as large as its caller had, and on Python 3.14 CPython is told where
that stack is, so its own recursion still ends in RecursionError rather than a crash.
$replace drives a regex or a matcher function as jsonata-js does
(#198). A replacement function's
$m.start and $m.end count UTF-16 code units, so $replace("éab", /a/, function($m) {$string($m.start)}) is "é1b" (it was "é2b"). A zero-length match after the first is
D1004 with a replacement function too, so $replace("abc", /x*/, function($m){"-"}) raises
it (it gave "-a-b-c-"), and is still found after the limit is reached. A first match may
be zero-length, so $replace("abc", /x*$/, "-") is "abc-" (it was D1004). A custom
matcher function works as the pattern, with a replacement string or function, and its
result is checked as $match checks it (T1010).
$$name is the variable named $name
(#201). jsonata-js reads everything after
the leading $ as the variable name, so $$ is the variable $ (the input) and $$a is
the variable $a, undefined unless bound. Over {"a": [1, 2, 3]}, $$a and $string($$a)
are undefined (they were null and "null"), ($a := 5; $$a) is undefined, and
($$x := 2; $$x * 3) is 6. $$ and $$.a are unchanged.
A lambda called as a path step gets only the arguments written
(#202). The parser no longer passes the
context item as an implicit first argument, so over {"a": [1, 2, 3]}
a.function($v) { $v * 2 }() is undefined (it was [2, 4, 6]), as in jsonata-js; pass
$ to get the items. A parenthesised step called the same way, as in
a.(function($v) { $v * 2 })($), now works (it was T1006).
Calling an expression applies any function value, as jsonata-js does. ($uppercase)("x")
is "X" and a.($string)() calls the builtin per item (both were errors). A ? argument
partially applies it, so (function($v, $w) { $v + $w })(?, 1)(5) is 6, also as the last
expression of a lambda ($f := function() { $g(?) } was an error). Calling a
non-function, as in (1)(2), raises T1006 (it had no code).
$ may appear inside a name, as in jsonata-js: $a$b is the variable a$b and a$b
the field a$b (both were parse errors).
@ on a block, * or $ outside a path binds nothing
(#213). jsonata-js only applies a
binding to a path step, so (acct.Order.P)@$v is the three products (it was the input,
three times), *@$v[0] is the first child and (acct.Order.P){N: N}@$v groups the
products. A ., sort or # after it still makes it a path. $[]#$i drops the [], as
the path # builds keeps no singleton array.
A % filter on a block or * outside a path raises S0217
(#211). jsonata-js reports
(acct.Order.P)[%.id = "o1"] and *[%.id = "o1"], which were accepted here. Inside a
path, as in (acct.Order.P)[%.id = "o1"].N, they still work. A % on either side of ~>
is not passed on, as in jsonata-js, so %.id ~> $count is 0 where it raised S0217.
A filtered group-by under a sort, as in (acct.Order.P)[%.id = "o1"]{N: N}^(N), gives a
value, and @ after a filtered group-by, as in acct.Order.P[0]{N: N}@$v, raises S0215.
A sort over a block's tuple stream unwraps a single or empty result
(#207). (acct.Order.P[N="C"])^(%.id)
is {"N":"C"} (it was [{"N":"C"}]), and (acct.Order.P)[N = "Z"]^(>%.id) is undefined
(it was []).
A % looks back through a group-by on a bare * or %
(#209). *{N: N}.% is the whole
input and acct.Order.P.(%{"x": id}).% reaches acct, as in jsonata-js, where both raised
S0217. Inside a block, as in acct.(*{"k": 1}).% or acct.(*).%, the % is undefined.
A group-by over an empty tuple stream is {}
(#215). acct.Order.P[%.id = "zz"]{"k": 1}
gave {"k": 1}. A group-by over an empty plain value still evaluates its pairs once, so
nope{"k": $count($)} stays {"k": 0}.
% inside a group-by no longer raises S0217
(#192). As in jsonata-js, a % in a
group-by's key or value is never resolved and evaluates to undefined, so
acct.Order.P{%.id: N} is {} and acct.Order.P{N: $count(%)} counts 0. A % after a
group-by, as in (acct.Order.P{N: N}).%, seeks through the grouped path. S0217 now uses
jsonata-js's message: "The object representing the 'parent' cannot be derived from this
expression".
~> follows jsonata-js evaluateApplyExpression
(#193). The left side is evaluated once
and passed on even when undefined, so nope ~> $count() is 0 and nope ~> $append([]) is
[] (both used to be undefined). A call on the right gets the left side as its first
argument. Any other right side is evaluated and must be a function: two functions compose
wherever they appear, so arr ~> ($sum ~> $string) and $type($sum ~> $string) now work, and
5 ~> nope raises T2006 instead of giving undefined. A partial application is applied like
any function, so "abc" ~> $substring(?, ?) is "abc". The left side used to be evaluated
twice, calling a host function on it twice. Two jsonata-js behaviours are deliberately not
followed: it drops the filter on a call (arr ~> $map($f)[0] gives the whole array there),
and it matches a regex against an undefined left side as the text "undefined".
A transform is always a function value that runs when invoked, as in jsonata-js. It used
to run at once wherever $ was bound, so a ~> into a transform inside another transform's
update, or a.(|x|{}|), ran it on the wrong input. $ in a transform's update is now the
matched object, not the transform's input, and so it is in the delete clause, which is
evaluated for each matched object as jsonata-js does (it used to be evaluated once, against
the input). A delete clause that is not a string or an array of strings raises T2012, as
does one with a non-string item, which used to be skipped, and a transform called with more
than one argument raises T0410. Like a lambda, it closes over the variables it
uses, and written directly as a higher-order function's argument it is called, as a
composition such as $map(n, $sum ~> $string) now is.
$sort, $single and the other higher-order builtins can be called by value:
[3,1,2] ~> $sort, $f := $single and $sort ~> $reverse used to raise T0410. They now run
as a direct call does, validating their arguments the same way.
A host function named without a call is a function value, as a builtin is, so
5 ~> $dbl, $f := $dbl, $map(arr, $dbl) and $exists($dbl) work for a function
registered with register. $exists on a variable checks its binding first, so a bound
name that shadows a builtin, such as ($sort := nope; $exists($sort)), is false.
A lambda parameter without an argument is undefined, not null, on every path:
($f := function($v) { $type($v) }; $f()) used to be "null" when the body was simple
enough to compile, and a tail-recursive lambda bound it as null too.
$single, transforms and partial applications handle undefined as jsonata-js does
(#193). $single gives undefined for an
undefined input and treats null as a one-item sequence. A transform gives undefined for an
undefined input and raises T0410 for one that is not an object or array, however it is
invoked ($t(5) used to return 5). A placeholder left without an argument is undefined, not
null.
?? propagates errors and coalesces any undefined value, as jsonata-js does
(#195). jsonata-js reads a ?? b as
$exists(a) ? a : b. $error("x") ?? 1 used to swallow the error and give 1, and the right
side could then see variables left bound by the failed left side. An undefined left side that
isn't a path or a variable, such as $lookup({}, "a") ?? 3, used to give undefined instead of
3.
Each evaluation runs in a scope of its own, as jsonata-js runs each in a new frame. An
Evaluator reused through the Rust API used to keep what an earlier evaluation bound: a
top-level $x := 1, and on a failed evaluation any scope or temporary $ an error skipped
popping. Bindings set on the Context beforehand are still visible. $$ is each evaluation's
own input; a reused evaluator used to keep answering it from the first.
A .(...) step's := bindings stay in the step. ($y := 1; a.($y := 5); $y) is 1, as
in jsonata-js; it used to be 5. In the Rust AST, a .(expr) step is now always
FunctionApplication(Block([...])), even with one expression, so it can be told apart from a
.$fn(...) step, whose := arguments still bind outside the step.
A builtin's arguments are all evaluated. Two leftover checks returned early: one for an
unbound variable argument, which skipped the other arguments' errors and signature checks and
gave null rather than undefined ({"x": $string($nope)} is now {}); and one that
evaluated a path argument a first time, discarding any error, before evaluating it again.
$match returns every match, as jsonata-js does (#194).
jsonata-js regexes are always global, so $match("aba", /a/) gives both matches, not only
the first, and the limit argument caps how many are returned. Each match object now always
has groups, and an unmatched capture group in it is undefined, not null. index counts
UTF-16 code units like JavaScript strings, not bytes, so it changes for text before the match
that isn't ASCII. A negative limit raises D3040, and a regex that would match a zero-length
string again raises D1004. A custom matcher's results are checked as jsonata-js checks them,
so a malformed result raises T1010, and a limit of 0 no longer calls the matcher.
A fractional limit in $match and $replace rounds up, as in jsonata-js, which keeps
matching while the count is below the limit. $replace("aaa", "a", "b", 1.5) is now "bba".
Field paths in lambda bodies map over arrays. ($f := function() { objs.p }; $f()) was
undefined when objs held an array; it is now [1, 2] as in jsonata-js.
Compiled predicates read the right field from objects with different key orders. A
predicate inside a lambda body, such as xs[a > 2].b, read fields by the first object's key
positions and so tested the wrong field on objects ordered differently.
Index binding on a sort numbers the sorted items: arr^($)#$i is the sorted array and
$i its positions, where it used to return the input object.
Object keys that are null or repeated raise jsonata-js's errors: a null key is T1003
(it used to be skipped), and two pairs with the same literal key are D1009 (the later
value used to win).
unknown(function) now reports T1006 ("Attempted to invoke a non-function") instead of
S0202. The function keyword in argument position used to commit the parser to a lambda,
which then failed on its shape before anything decided the call target was not a function.
(reference suite errors/case005, #150)
A choice group containing a parameterized type — <(sa<n>)> — now reports S0402
("Choice groups containing parameterized types are not supported") instead of S0202. The
signature parser rejected the shape while still reading characters, so it never got far enough
to say what it was looking at. S0401 was carried as text inside an uncoded error and is now
a first-class variant too.
(reference suite function-signatures/case034,
#150)

With these two, the jsonata-js reference suite is at 1686/1686 and
KNOWN_DIVERGENCES is empty.

Full commit list: v2.2.11...v2.2.12

Test Suite Compatibility

See the README for the current
reference-suite pass count -- this changelog no longer hardcodes a number here, since a
stale hardcoded count is exactly the kind of bug this project has previously shipped
and had to correct.

Installation

pip install jsonatapy==2.2.12

# Cargo.toml
jsonata-core = "2.2.12"

Verification

import jsonatapy
print(jsonatapy.__version__)  # Should print: 2.2.12

Navigation Menu

Repository navigation

Release v2.2.12 - "Gymkata"

Choose a tag to compare

Sorry, something went wrong.