Skip to content

futile/jtc

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

jtc - cli tool to extract, manipulate and transform source JSON

jtc stand for: JSON transformational chains (used to be JSON test console).

jtc offers a powerful way to select one or multiple elements from a source JSON and apply various actions on the selected elements at once (wrap selected elements into a new JSON, filter in/out, sort elements, update elements, insert new elements, remove, copy, move, compare, transform, swap around and many other operations).

Content:

  1. Short description

  2. Compilation and installation options

  3. Quick-start guide

  4. Complete User Guide

  5. Class usage and c++14 interface

  6. jtc vs jq

Short description

- jtc is a simple yet very powerful and efficient cli utility tool to process and manipulate JSON data

jtc offers following features (a short list of main features):

  • simple user interface allowing applying a bulk of changes in a single or chained sets of commands
  • featured walk-path interface lets extracting any combination of data from sourced JSON trees
  • extracted data is representable either as found, or could be encapsulated in JSON array/object or transfored using templates
  • support Regular Expressions when searching source JSON
  • fast and efficient processing of very large JSON files (various built-in search caches)
  • insert/update operations optionally may undergo shell cli evaluation
  • support in-place modifications of the input/source JSON file
  • features namespaces, facilitating interpolation of preserved JSON values in templates
  • supports buffered and streamed modes of input read
  • sports concurent input JSON reading/parsing (on multi-core CPU)
  • written entirely in C++14, no dependencies (STL only, idiomatic C++, no memory leaks)
  • extensively debuggable
  • conforms JSON specification (json.org)

The walk-path feature is easy to understand - it's only made of 2 kinds of lexemes:

  • subscripts - enclosed into [, ], subscripts let traversing JSON tree downwards (towards the leaves) and upwards (towards the root)
  • search lexemes - encased as <..> or >..< (for a recursive and non-recursive searches respectively); search lexemes facilitate various match criteria defined by an optional suffix and/or quantifier

Both kinds of lexemes cab be iterable:

  • iterable subscripts let iterating over children of currently addressed JSON iterables nodes (arrays/objects),
  • while iterable search lexemes let iterating over all (recursive) matches for a given search criteria

A walk-path may have an arbitrary number of lexemes -the tool accepts a virtually unlimited number of walk paths. See below more detailed explanation with examples

Compilation and installation options

For compiling, c++14 (or later) is required. To compile under different platforms:

  • MacOS/BSD: c++ -o jtc -Wall -std=c++14 -Ofast jtc.cpp
  • Linux:
    • non-relocatable (dynamically linked) image:
      • c++ -o jtc -Wall -std=gnu++14 -Ofast -pthread -lpthread jtc.cpp
    • relocatable (statically linked) image:
      • c++ -o jtc -Wall -std=gnu++14 -Ofast -static -Wl,--whole-archive -lrt -pthread -lpthread -Wl,--no-whole-archive jtc.cpp
  • Debian: c++ -o jtc -Wall -std=c++14 -pthread -lpthread -Ofast jtc.cpp (ensure c++ poits to clang++-6.0 or above)

pass -DNDEBUG flag if you like to compile w/o debugs, however it's unadvisable - there's no performance gain from doing so

Linux and MacOS precompiled binaries are available for download

or download the latest precompiled binary:

Packaged installations:

Installing via MacPorts

On MacOS, you can install jtc via the MacPorts package manager:

$ sudo port selfupdate
$ sudo port install jtc

Installation on Linux distributions

jtc is packaged in the following Linux distributions and can be installed via the package manager.

  • Fedora: jtc is present in Fedora 31 and later:
$ dnf install jtc
  • openSUSE: jtc can be installed on openSUSE Tumbleweed via zypper:
$ zypper in jtc

or on Leap 15.0 and later by adding the utilities repository and installing jtc via zypper.

Compile and install instructions:

download jtc-master.zip, unzip it, descend into unzipped folder, compile using an appropriate command, move compiled file into an install location.

here're the example steps for MacOS:

  • say, jtc-master.zip has been downloaded to a folder and the terminal app is open in that folder:
  • unzip jtc-master.zip
  • cd jtc-master
  • c++ -o jtc -Wall -std=c++17 -Ofast jtc.cpp
  • sudo mv ./jtc /usr/local/bin/

Release Notes

See the latest Release Notes

Quick-start guide:

run jtc -g for walk-path explanations, usage notes and additional usage examples

Consider a following JSON (a mockup of a bookmark container), stored in a file Bookmarks:

{
   "Bookmarks": [
      {
         "children": [
            {
               "children": [
                  { "name": "The New York Times", "stamp": "2017-10-03, 12:05:19", "url": "https://www.nytimes.com/" },
                  { "name": "HuffPost UK", "stamp": "2017-11-23, 12:05:19", "url": "https://www.huffingtonpost.co.uk/" }
               ],
               "name": "News",
               "stamp": "2017-10-02, 12:05:19"
            },
            {
               "children": [
                  { "name": "Digital Photography Review", "stamp": "2017-02-27, 12:05:19", "url": "https://www.dpreview.com/" }
               ],
               "name": "Photography",
               "stamp": "2017-02-27, 12:05:19"
            }
         ],
         "name": "Personal",
         "stamp": "2017-01-22, 12:05:19"
      },
      {
         "children": [
            { "name": "Stack Overflow", "stamp": "2018-05-01, 12:05:19", "url": "https://stackoverflow.com/" },
            { "name": "C++ reference", "stamp": "2018-06-21, 12:05:19", "url": "https://en.cppreference.com/" }
         ],
         "name": "Work",
         "stamp": "2018-03-06, 12:07:29"
      }
   ]
}

1. let's start with a simple thing - list all URLs:

bash $ jtc -w'<url>l:' Bookmarks
"https://www.nytimes.com/"
"https://www.huffingtonpost.co.uk/"
"https://www.dpreview.com/"
"https://stackoverflow.com/"
"https://en.cppreference.com/"

Let's take a look at the walk-path <url>l::

  • search lexemes are enclosed in angular brackets <, > - that style provides a recursive search throughout JSON
  • suffix l instructs to search among labels only
  • quantifier : instructs to find all occurrences, such quantifiers makes a path iterable

2. dump all bookmark names from the Work folder:

bash $ jtc -w'<Work>[-1][children][:][name]' Bookmarks
"Stack Overflow"
"C++ reference"

Here the walk-path <Work>[-1][children][:][name] is made of following lexemes:

a. <Work>: find within a JSON tree the first occurrence where the JSON string value is matching "Work" exactly
b. [-1]: step up one tier in the JSON tree structure (i.e., address an immediate parent of the found JSON element)
c. [children]: select/address a node whose label is "children" (it'll be a JSON array, at the same tier with Work)
d. [:]: select each node in the array e. [name]: select/address a node with the label "name"

in order to understand better how the walk-path works, let's run that series of cli in a slow-motion, gradually adding lexemes to the path one by one, perhaps with the option -l to see also the labels (if any) of the selected elements:

bash $ jtc -w'<Work>' -l Bookmarks
"name": "Work"
bash $ jtc -w'<Work>[-1]' -l Bookmarks
{
   "children": [
      {
         "name": "Stack Overflow",
         "stamp": "2018-05-01, 12:05:19",
         "url": "https://stackoverflow.com/"
      },
      {
         "name": "C++ reference",
         "stamp": "2018-06-21, 12:05:19",
         "url": "https://en.cppreference.com/"
      }
   ],
   "name": "Work",
   "stamp": "2018-03-06, 12:07:29"
}
bash $ jtc -w'<Work>[-1][children]' -l Bookmarks
"children": [
   {
      "name": "Stack Overflow",
      "stamp": "2018-05-01, 12:05:19",
      "url": "https://stackoverflow.com/"
   },
   {
      "name": "C++ reference",
      "stamp": "2018-06-21, 12:05:19",
      "url": "https://en.cppreference.com/"
   }
]
bash $ jtc -w'<Work>[-1][children][:]' -l Bookmarks
{
   "name": "Stack Overflow",
   "stamp": "2018-05-01, 12:05:19",
   "url": "https://stackoverflow.com/"
}
{
   "name": "C++ reference",
   "stamp": "2018-06-21, 12:05:19",
   "url": "https://en.cppreference.com/"
}
bash $ jtc -w'<Work>[-1][children][:][name]' -l Bookmarks
"name": "Stack Overflow"
"name": "C++ reference"

3. dump all URL's names:

bash $ jtc -w'<url>l:[-1][name]' Bookmarks
"The New York Times"
"HuffPost UK"
"Digital Photography Review"
"Stack Overflow"
"C++ reference"

this walk-path <url>l:[-1][name]:

  • finds recursively (encasement <, >) each (:) JSON element with a label (l) matching url
  • then for an each found JSON element, select its parent ([-1])
  • then, select a JSON (sub)element with the label "name"

4. dump all the URLs and their corresponding names, preferably wrap found pairs in JSON:

bash $ jtc -w'<url>l:' -w'<url>l:[-1][name]' -jl Bookmarks
[
   {
      "name": "The New York Times",
      "url": "https://www.nytimes.com/"
   },
   {
      "name": "HuffPost UK",
      "url": "https://www.huffingtonpost.co.uk/"
   },
   {
      "name": "Digital Photography Review",
      "url": "https://www.dpreview.com/"
   },
   {
      "name": "Stack Overflow",
      "url": "https://stackoverflow.com/"
   },
   {
      "name": "C++ reference",
      "url": "https://en.cppreference.com/"
   }
]
  • yes, multiple walks (-w) are allowed
  • option -j will wrap the walked outputs into a JSON array, but not just,
  • option -l used together with -j will ensure relevant walks are grouped together (try without -l)
  • if multiple walks (-w) are present, by default, walked results will be printed interleaved (if it can be interleaved)

5. Subscripts (offsets) and Searches explained

In short:

  • Subscript lexemes ([..]) facilitate:
    • addressing children (by index/label) in JSON iterables (arrays and objects) - i.e., traverse JSON structure downward from the root (toward leaves), e.g.: [2], [id]
    • addressing parents (immediate and distant) - i.e., traverse JSON structure upwards, toward the the root (from leaves), e.g.: [-1] (tier offset from the currently walked/found element), [^2] (tier offset from the root towards walked/found element)
    • select ranges and slices of JSON elements in JSON iterables, e.g.: [+2], [:], [:3], [-2:], [1:-1:2]
  • Search lexemes (<..>, >..<) facilitate:
    • recursive (<..>) and non-recursive (>..<) matches
    • there're optional one-letter suffixes that may follow the lexemes (e.g.: <..>Q) which define type of search: (REGEX) string search, (REGEX) label search, (REGEX) numerical, boolean, null, atomic, objects, arrays (or either), arbitrary JSONs, unique, duplicates, sorted match, etc.
    • there're also optional quantifiers to search lexemes (must take the last position in the search lexeme, after the suffix if one present) - let selecting match instance, or range of matches (e.g.: <id>l3- will match 4th (zero based) label "id"; if no quantifier present 0 is assumed - first match)
  • a subscript lexeme could be grouped with a search lexeme over ':' to facilitate a scoped search, e.g.: [id]:<value> is a single lexeme which will match recursively the first occurrence of the string "value" with the label "id" - i.e., "id": "value"
  • Directives: there are a few suffixes which turn a search lexeme into a directive:
    • directives do not do any matching, instead they facilitate a certain action/operation with the currently walked JSON element, like: memorize it in the namespace, or erase from it, or memorize its label, or perform a shell cli evaluation, etc
    • couple directives (<>f and <>F) facilitate also walk branching, jumping walks and walk reiterations

Refer to jtc User Guide for the detailed explanation of the subscripts, search lexemes and directives.

6. Debugability / JSON validation

jtc is extensively debuggable: the more times option -d is passed the more debugs will be produced. Enabling too many debugs might be overwhelming, though one specific case many would find extremely useful - when validating a failing JSON:

bash $ <addressbook-sample.json jtc 
jtc json exception: expected_json_value

If JSON is big, it's desirable to locate the parsing failure point. Passing just one -d let easily spotting the parsing failure point and its locus:

bash $ <addressbook-sample.json jtc -d
.display_opts(), option set[0]: -d (internally imposed: )
.init_inputs(), reading json from <stdin>
.exception_locus_(), ...         }|       ],|       "children": [,],|       "spouse": null|    },|    {|  ...
.exception_spot_(), -------------------------------------------->| (offset: 967)
jtc json parsing exception (<stdin>:967): expected_json_value
bash $ 

Complete User Guide

Refer to a complete User Guide for further examples and guidelines.

A tiny example of class usage and its interface (c++14):

Say, we want to accomplish a following task:

  1. read Address Book JSON from <stdin>
  2. sort all records by Name (for simplicity, assume all records have that label)
  3. output resulting Address Book JSON

Below is the code sample how that could be achieved using Json.hpp class and the source JSON - Address Book:

#include <iostream>
#include <fstream>
#include "lib/Json.hpp"

// compile with: c++ -o sort_ab -Wall -std=c++14 sorting_ab.cpp
using namespace std;


int main(int argc, char *argv[]) { 
 // read and parse json from cin:
 Json jin{ {istream_iterator<char>{cin>>noskipws}, istream_iterator<char>{}} };
 
 // get all the names into vector and sort them
 vector<string> names{jin.walk("<Name>l:"), jin.walk().end()};
 sort(names.begin(), names.end());

 // rebuild AB with sorted records:
 Json sorted = ARY{};
 for(const auto &name: names)
  sorted.push_back(move( *jin.walk("[Name]:<" + name + ">[-1]") ));

 // // put back into the original container and print using indentation 3 
 jin["AddressBook"].clear().push_back( move(sorted) ); 
 cout << jin.tab(3) << endl;                                    // default tab is 1
}

Address Book JSON:

bash $ jtc -tc addressbook-sample.json 
{
   "AddressBook": [
      {
         "Name": "John",
         "address": { "city": "New York", "postal code": 10012, "state": "NY", "street address": "599 Lafayette St" },
         "age": 25,
         "children": [ "Olivia" ],
         "phoneNumbers": [
            { "number": "212 555-1234", "type": "mobile" },
            { "number": "213 123-2368", "type": "mobile" }
         ],
         "spouse": "Martha"
      },
      {
         "Name": "Ivan",
         "address": { "city": "Seattle", "postal code": 98104, "state": "WA", "street address": "5423 Madison St" },
         "age": 31,
         "children": [],
         "phoneNumbers": [
            { "number": "573 923-6483", "type": "home" },
            { "number": "523 283-0372", "type": "mobile" }
         ],
         "spouse": null
      },
      {
         "Name": "Jane",
         "address": { "city": "Denver", "postal code": 80206, "state": "CO", "street address": "6213 E Colfax Ave" },
         "age": 25,
         "children": [ "Robert", "Lila" ],
         "phoneNumbers": [
            { "number": "358 303-0373", "type": "office" },
            { "number": "333 638-0238", "type": "home" }
         ],
         "spouse": "Chuck"
      }
   ]
}
bash $ 

Output result:

bash $ <addressbook-sample.json sort_ab | jtc -tc      # using jtc here only for a compact view
{
   "AddressBook": [
      [
         {
            "Name": "Ivan",
            "address": { "city": "Seattle", "postal code": 98104, "state": "WA", "street address": "5423 Madison St" },
            "age": 31,
            "children": [],
            "phoneNumbers": [
               { "number": "573 923-6483", "type": "home" },
               { "number": "523 283-0372", "type": "mobile" }
            ],
            "spouse": null
         },
         {
            "Name": "Jane",
            "address": { "city": "Denver", "postal code": 80206, "state": "CO", "street address": "6213 E Colfax Ave" },
            "age": 25,
            "children": [ "Robert", "Lila" ],
            "phoneNumbers": [
               { "number": "358 303-0373", "type": "office" },
               { "number": "333 638-0238", "type": "home" }
            ],
            "spouse": "Chuck"
         },
         {
            "Name": "John",
            "address": { "city": "New York", "postal code": 10012, "state": "NY", "street address": "599 Lafayette St" },
            "age": 25,
            "children": [ "Olivia" ],
            "phoneNumbers": [
               { "number": "212 555-1234", "type": "mobile" },
               { "number": "213 123-2368", "type": "mobile" }
            ],
            "spouse": "Martha"
         }
      ]
   ]
}
bash $ 

for the complete description of Json class interface, refer to Json.hpp

Btw, the same sorting is achievable using <>g lexeme:

bash $ <addressbook-sample.json jtc -tc -w'[0]' -pi'[Name]:<>g:[-1]'
{
   "AddressBook": [
      {
         "Name": "Ivan",
         "address": { "city": "Seattle", "postal code": 98104, "state": "WA", "street address": "5423 Madison St" },
         "age": 31,
         "children": [],
         "phoneNumbers": [
            { "number": "573 923-6483", "type": "home" },
            { "number": "523 283-0372", "type": "mobile" }
         ],
         "spouse": null
      },
      {
         "Name": "Jane",
         "address": { "city": "Denver", "postal code": 80206, "state": "CO", "street address": "6213 E Colfax Ave" },
         "age": 25,
         "children": [ "Robert", "Lila" ],
         "phoneNumbers": [
            { "number": "358 303-0373", "type": "office" },
            { "number": "333 638-0238", "type": "home" }
         ],
         "spouse": "Chuck"
      },
      {
         "Name": "John",
         "address": { "city": "New York", "postal code": 10012, "state": "NY", "street address": "599 Lafayette St" },
         "age": 25,
         "children": [ "Olivia" ],
         "phoneNumbers": [
            { "number": "212 555-1234", "type": "mobile" },
            { "number": "213 123-2368", "type": "mobile" }
         ],
         "spouse": "Martha"
      }
   ]
}
bash $

jtc vs jq:

jtc was inspired by the complexity of jq interface (and its DSL), aiming to provide a user tool which would let attaining the desired result in a more feasible and succinct way

utility ideology:

  • jq is a stateful processor with own DSL, variables, operations, control flow logic, IO system, etc, etc
  • jtc is a unix utility confining its functionality to operation types with its data model only (as per unix ideology). jtc performs one major operation at a time (like insertion, update, swap, etc), however multiple operations could be chained using / delimiter

jq is non-idiomatic in a unix way, e.g.: one can write a program in jq language that even has nothing to do with JSON. Most of the requests (if not all) to manipulate JSONs are ad hoc type of tasks, and learning jq's DSL for ad hoc type of tasks is an overkill (that purpose is best facilitated with GPL).
The number of asks on the stackoverflow to facilitate even simple queries for jq is huge - that's the proof in itself that for many people feasibility of attaining their asks with jq is a way too low, hence they default to posting their questions on the forum.

jtc on the other hand is a utility (not a language), which employs a novel but powerful concept, which "embeds" the ask right into the walk-path. That facilitates a much higher feasibility of attaining a desired result: building a walk-path a lexeme by lexeme, one at a time, provides an immediate visual feedback and let coming up with the desired result rather quickly.

learning curve:

  • jq: before you could come up with a query to handle even a relatively simple ask, you need to become an expert in jq language, which will take some time. Coming up with the complex queries requires what it seems having a PhD in jq, or spending lots of time on stackoverflow and similar forums
  • jtc employs only a simple (but powerful) concept of the walk-path (which is made only of 2 types of lexemes, each type though has several variants) which is quite easy to grasp.

handling irregular JSONs:

  • jq: handling irregular JSONs for jq is not a challenge, building a query is! The more irregularities you need to handle the more challenging the query (jq program) becomes
  • jtc was conceived with the idea of being capable of handling complex irregular JSONs with a simplified interface - that all is fitted into the concept of the walk-path, while daisy-chaining multiple operations is possible to satisfy almost every ask.

solutions input invariance

- most of jtc solutions would be input invariant (hardly the same could be stated for jq). Not that it's impossible to come up with invariant solutions in jq, it's just a lot more harder, while jtc with its walk-path model prompts for invariant solutions. I.e., the invariant solution will keep working even once the JSON outer format changes (the invariant solution only would stop working once the relationship between walked JSON elements changes).
E.g.: consider a following query, extract format [ "name", "surname" ] from 2 types of JSON:

bash $ case1='{"Name":"Patrick", "Surname":"Lynch", "gender":"male", "age":29}'
bash $ case2='[{"Name":"Patrick", "Surname":"Lynch", "gender":"male", "age":29},{"Name":"Alice", "Surname":"Price", "gender":"female", "age":27}]'

a natural, idiomatic jtc solution would be:

bash $ <<<$case1 jtc -w'<Name>l:<N>v[-1][Surname]' -rT'[{{N}},{{}}]'
[ "Patrick", "Lynch" ]
bash $ <<<$case2 jtc -w'<Name>l:<N>v[-1][Surname]' -rT'[{{N}},{{}}]'
[ "Patrick", "Lynch" ]
[ "Alice", "Price" ]

While one of the most probable jq solution would be:

bash $ <<<$case1 jq -c 'if type == "array" then .[] else . end | [.Name, .Surname]'
["Patrick","Lynch"]
bash $ <<<$case2 jq -c 'if type == "array" then .[] else . end | [.Name, .Surname]'
["Patrick","Lynch"]
["Alice","Price"]

The both solutions work correctly, however, any change in the outer encapsulation will break jq's solution , while jtc will keep working even if JSON is reshaped into an irregular structure, e.g.:

#jtc:
bash $ case3='{"root":[{"Name":"Patrick", "Surname":"Lynch", "gender":"male", "age":29}, {"closed circle":[{"Name":"Alice", "Surname":"Price", "gender":"female", "age":27}, {"Name":"Rebecca", "Surname":"Hernandez", "gender":"female", "age":28}]}]}'
bash $ <<<$case3 jtc -w'<Name>l:<N>v[-1][Surname]' -rT'[{{N}},{{}}]'
[ "Patrick", "Lynch" ]
[ "Alice", "Price" ]
[ "Rebecca", "Hernandez" ]

#jq:
bash $ <<<$case3 jq -c 'if type == "array" then .[] else . end | [.Name, .Surname]'
[null,null]

The same property makes jtc solutions resistant to cases of incomplete data, e.g.: if we drop "Name" entry from one of the entries in case 2, jtc solution still works correctly:

#jtc:
bash $ case2='[{"Surname":"Lynch", "gender":"male", "age":29},{"Name":"Alice", "Surname":"Price", "gender":"female", "age":27}]'
bash $ <<<$case2 jtc -w'<Name>l:<N>v[-1][Surname]' -rT'[{{N}},{{}}]'
[ "Alice", "Price" ]

#jq:
bash $ <<<$case2 jq -c 'if type == "array" then .[] else . end | [.Name, .Surname]'
[null,"Lynch"]
["Alice","Price"]

- i.e., jtc will not assume that user would require some default substitution in case of incomplete data (but if such handling is required then the walk-path can be easily enhanced)

programming model

  • jq is written in C, which drags all intrinsic problems the language has dated its creation (here's what I mean)
  • jtc is written in the idiomatic C++14 using STL only. jtc does not have a single naked memory allocation operator (those few new operators required for legacy interface are implemented as guards), nor it has a single naked pointer acting as a resource holder/owner, thus jtc is guaranteed to be free of memory/resourses leaks (at least one class of the problems is off the table) - STL guaranty.
    Also, jtc is written in a very portable way, it should not cause problems compiling it under any unix like system.

JSON numerical fidelity:

  • jq is not compliant with JSON numerical definition. What jq does, it simply converts a symbolic numerical representation to an internal binary and keeps it that way. That approach:
    • is not compliant with JSON definition of the numerical values
    • it has problems retaining required precision
    • might change original representation of numericals
    • leads to incorrect processing of some JSON streams
  • jtc validates all JSON numericals per JSON standard and keep numbers internally in their original literal format, so it's free of all the above caveats, compare:
Handling jtc jq 1.6
Invalid Json: [00] <<<'[00]' jtc <<<'[00]' jq -c .
Parsing result jtc json parsing exception (<stdin>:3): missed_prior_enumeration [0]
Precision test: <<<'[0.99999999999999999]' jtc -r <<<'[0.99999999999999999]' jq -c .
Parsing result [ 0.99999999999999999 ] [1]
Retaining original format: <<<'[0.00001]' jtc -r <<<'[0.00001]' jq -c .
Parsing result [ 0.00001 ] [1e-05]
Stream of atomic JSONs: <<<'{}[]"bar""foo"00123truefalsenull' jtc -Jr <<<'{}[]"bar""foo"00123truefalsenull' jq -sc
Parsing result [ {}, [], "bar", "foo", 0, 0, 123, true, false, null ] parse error: Invalid numeric literal at line 2, column 0

performance:

  • jq is a single-threaded process
  • jtc engages a concurent (multi-threaded) reading/parsing when multiple files given (the advantage could be observed on multi-core CPU, though it become noticeable only with relatively big JSONs or with relatively big number of files processed)

Comparison of single-threaded performance: here's a 4+ million node JSON file standard.json:

bash $ time jtc -zz standard.json 
4329975
user 5.537 sec

The table below compares jtc and jq performance for similar operations (using TIMEFORMAT="user %U sec"):

jtc 1.75 jq 1.6
parsing JSON: parsing JSON:
bash $ time jtc -t2 standard.json | md5 bash $ time jq -M . standard.json | md5
d3b56762fd3a22d664fdd2f46f029599 d3b56762fd3a22d664fdd2f46f029599
user 8.679 sec user 19.570 sec
removing by key from JSON: removing by key from JSON:
bash $ time jtc -t2 -pw'<attributes>l:' standard.json | md5 bash $ time jq -M 'del(..|.attributes?)' standard.json | md5
0624aec46294399bcb9544ae36a33cd5 0624aec46294399bcb9544ae36a33cd5
user 9.442 sec user 28.624 sec
updating JSON recursively by label: updating JSON recursively by label:
bash $ time jtc -t2 -w'<attributes>l:[-1]' -i'{"reserved": null}' standard.json | md5 bash $ time jq -M 'walk(if type == "object" and has("attributes") then . + { "reserved" : null } else . end)' standard.json | md5
6c86462ae6b71e10e3ea114e86659ab5 6c86462ae6b71e10e3ea114e86659ab5
user 12.292 sec user 30.255 sec

Machine spec used for testing:

  Model Name:                 MacBook Pro
  Model Identifier:           MacBookPro15,1
  Processor Name:             Intel Core i7
  Processor Speed:            2,6 GHz
  Number of Processors:       1
  Total Number of Cores:      6
  L2 Cache (per Core):        256 KB
  L3 Cache:                   12 MB
  Hyper-Threading Technology: Enabled
  Memory:                     16 GB 2400 MHz DDR4

compare jtc based solutions with jq's:

Here are published some answers for JSON queries using jtc, you may compare those with jq's, as well as study the feasibility of the solutions, test relevant peroformance, etc

Refer to a complete User Guide for further examples and guidelines.

Enhancement requests are more than welcome: ldn.softdev@gmail.com

About

JSON processing utility

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C++ 100.0%