A streaming JSON pull parser in C.
Does the world really need another JSON parser? Maybe not. But what distinguishes this one is the desire to let you read through gigantic files larger than you can comfortably fit in memory and still find the parts you need.
It builds a parse tree, but you can tear down as much of the tree as you don't need while it is still in the middle of parsing.
You can create a
json_pull parser object
json_begin_file() to begin reading from a file
json_begin_string() to begin reading from a string.
You can also read from anything you want by calling
with your own
read() function that copies UTF-8 bytes
from your input stream.
Reading full JSON trees
The simplest form is
json_read_tree(), which reads a complete
JSON object from the stream, or NULL if there was an error or on end of file.
json_free() on the object when you are done with it.
You can continue calling
json_read_tree() to read additional objects
from the same stream. This is not standard JSON, but is useful for something like
the Twitter filter stream that contains a series of JSON objects separated by
newlines, without a wrapper array that contains them all. (The previous parse tree
will automatically have
json_free() called on it to prevent memory leaks, unless
you have called
json_disconnect() to disconnect it from the parser.)
In addition, extra commas at the top level are no longer flagged as an error, so a series of comma-separated items can be read without their container array.
Reading JSON streams
With something like GeoJSON, where it is common to have a large wrapper array that contains many smaller items that are often useful to consider separately, you can read the stream one token at a time.
The normal form is
json_read(), which returns the next complete
object from the stream. This can be a single string, number,
null, or it can be an array or hash that
contains other primitive or compound objects.
Note that each array or hash will be returned following all the objects that it contains.
The outermost object will be the same one that
have returned, and you can tell that it is the outer object because its
parent field is null.
You can call
json_free() on each object as you are finished with it,
or wait until the end and call
json_free() on the outer object
which will recursively free everything that it contains. Freeing an object before
its container is complete also removes it from its parent array or hash so that
there are not dangling references left to it.
If you want to keep an individual JSON object around for later processing
while allowing its parents and siblings to be freed,
will detach an object from its container. You should call
later to free the detached object once you are finished with it.
Reading JSON streams with callbacks
If you are outputting a new stream as you read instead of just looking for the
sub-objects that interest you, you also need to know when arrays and hashes begin,
not just when they end, so you can output the opening bracket or brace. For this
purpose there is an additional streaming reader function,
json_read_separators(), which takes an additional argument for
a function to call when brackets, braces, commas, and colons are read.
Other object types and the closing of arrays and hashes are still sent through
the normal return value.
The types that can be sent to the callback function are
If there was an error while parsing, the parser will have returned NULL before
all the containers were closed. You will probably want to call
root elemement of the parser, which should contain the full parse
tree so far, to avoid leaking memory.
To free the parser object, call
json_end() on it.
JSON objects are represented as the C struct
It contains a
type field to indicate its type, a
pointer to the container that encloses it, and additional fields per type.
have no additional data.
Strings have type
JSON_STRING, with null-terminated UTF-8 text
string and length in
Numbers have type
JSON_NUMBER, with value in
and also preserve the original representation of the number
string and length in
Arrays have type
JSON_ARRAY. There are
length elements in the array,
and the elements are in
Hashes have type
JSON_HASH. There are
length key-value pairs,
and the keys are in
keys and the values in
The parser object has two fields of public interest:
error is a string
describing any errors found in the JSON, and
line is the current line number
being read from the input to make it easier to find the errors.
root field points to the outer object of the current parse tree.
There is a function
json_hash_get() that looks up the JSON object hash value
corresponding to a C string hash key in a JSON hash object. If the object specified is
NULL or not a JSON hash or has no matching key, it returns NULL.
json_stringify() converts a JSON parse tree back to its
string representation. The caller is responsible for calling
free() on the
return value once it is no longer needed.
jsoncat program reads JSON from the standard input or from
named files and pretty-prints it to the standard output.
Normally it uses the callback interface to avoid memory overhead. It has three options to exercise different aspects of the library:
-treads the whole tree and then prints it.
-ireads incrementally, but still keeps it all in memory.
-sreads the file into a string before parsing it with callbacks.
GeoJSON feature extraction
geojson2nd program reads JSON from the standard input input or from named files
and outputs each object where
object.type === "Feature' to the standard output on
a separate line. The resulting stream can then conveniently be used as input to
the ndjson-cli tools.