Skip to content

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 01 Oct 04:19
· 55 commits to main since this release

qj 0.2.0 is a rewrite of its core: qj now is jq 1.8.1, ported to Rust. jq's value and
number model, parser, compiler, bytecode VM, C builtins and builtin.jq run as jq's own
code does. qj's speed now comes from its input layer, which keeps jq's results exact:

  • simdjson parsing;
  • memory-mapped and streaming input;
  • ordered parallel processing of NDJSON;
  • evaluating simple filters directly on simdjson's tape.

Conformance

Conformance is checked byte for byte against the jq 1.8.1 binary: stdout, stderr and exit
code. The checks cover jq's own test suites, a corpus of about 39,000 cases (every builtin,
CLI option, input edge case and error path) and differential fuzzing, on macOS and Linux.

  • Default mode matches jq on every case except qj's own --help/--version text.
    Where jq crashes or hangs, qj gives a sane answer instead.
  • QJ_JQ_COMPAT=1 means "be exactly jq 1.8.1, bugs included", and matches on every
    case, with nothing normalized:
    • jq's name, help and version text;
    • jq's crashes, with stack overflows at jq's own depth per OS, and its hangs;
    • qj's extras turned off (globbing, .gz/.zst, --threads).
  • jq's own test suites pass 2,903/2,903, byte-exact.

What still differs, and why, is in docs/COMPATIBILITY.md.

Behaviour changes from 0.1.x

Output now matches jq exactly where 0.1.x differed:

  • number formatting follows jq 1.8.1's decNumber literals, so 1.10 stays 1.10 and
    1e2 prints as 1E+2;
  • string escapes are jq's (\u001b, not the input's \u001B);
  • error messages, exit codes, and the interleaving of stdout and stderr are jq's.

Other changes:

  • QJ_JQ_COMPAT=1 changed meaning, as above. Arithmetic now always follows jq's
    precision; it is no longer a separate switch.
  • The old evaluator and its NDJSON fast paths are gone, and so are QJ_CORE and
    QJ_NO_FAST_PATH. --jsonl and --debug-timing are still accepted and do nothing.
  • On Apple Silicon, qj uses every non-efficiency core, so Super and Performance cores on
    M5 Pro/Max.

Performance

Measured against 0.1.4 on the same M5 Max:

0.1.4 0.2.0
NDJSON 1.2 GB, -c . 681 ms 77 ms
NDJSON, {type, repo: .repo.name, actor: .actor.login} 147 ms 57 ms
JSON 49 MB, pretty print 192 ms 48 ms
JSON, group_by / sort_by / unique 135–137 ms 61–64 ms
reduce building a 50,000-key object 10.3 s 10 ms
peak memory, 3.4 GB NDJSON 3.5–3.8 GB 100–160 MB

Some cases are slower than in 0.1.4, though still well ahead of jq:

  • single-threaded select(...) over raw NDJSON, which 0.1.4 answered by scanning bytes
    and copying lines through unchanged, so its output wasn't jq's;
  • tight loops on the VM, such as limit/repeat.

See the README for comparisons with jq, jaq and
gojq.

Platforms

Prebuilt binaries for macOS (arm64, x86_64) and Linux (x86_64, aarch64), or
cargo install qj. Windows isn't supported.

Full changelog: v0.1.4...v0.2.0