Relaxed JSON5 parser

This is a relaxed and extended JSON5 parser that is heavily inspired by the great work of Dave Chaney pkg/json and his article about Building a high-performance JSON parser.

Warning: This work is not meant as a drop in replacement for the default encoding/json parser, and even if it provides a compatible interface it produces many essential but also subtle differences.

I have undertaken this journey in a trial to make it fit for fast parsing of very short default configuration scripts and tags out of pure curiosity and the lack of working alternatives in go. While the parser is now nearly production ready, I am not sure yet where this journey ends.

It currently supports the full JSON5 specification with braces, brackets, commas, colons, and all other JSON5 features, including:

• Support for unquoted keys in objects.
• Support for escape sequences in strings, including \n, \t, \\, etc.
• Support for single-line (//) and multi-line (/* ... */) comments.
• Support for integer, decimal, and hexadecimal numbers (e.g., 0x1E).
• Support for Unicode escape sequences in strings (e.g., \u{1F600}, \U0X1F4A9).

Besides, the parser also supports following extra features:

• Support for unquoted string values in objects and arrays, that are not reserved keywords (true, false, null, NaN, Infinity), and do not contain any leading or trailing whitespace or special characters.
• Support for complex numbers in values (e.g., 1+2i).

Note: Currently, the parser is not capable to switch between the strict JSON/JSON5 parsing modes, and the relaxed/extended JSON5 parsing modes, but this feature will be available in the near future (see Future plans).

At the moment I'm just more focusing on performance and correctness of the Scanner, Parser, and Decoder, as well as on a first design of the public interfaces. I must admit that I currently lack the vision of how to expose the different parsing modes and features I have in mind in a clean way.

Architecture

The architecture of the JSON parser is based on the following high level components:

• The Reader abstraction allows to read the JSON input data either from an io.Reader or directly from a static buffer while tracking the current scanner position. When using the io.Reader, the Reader can work together in two different modes:

  1. In a breathing mode (default), where it keeps a dynamic input buffer primarily containing the current token and its surrounding context, and
  2. In a growing mode where, it keeps the entire input data in memory and allows to access any part of it at any time.

• The Scanner abstraction allows to efficiently scan the input data provided by the Reader into tokens. The Scanner is coming in two main flavors with and without tracking of line and character position, as well as in multiple sub flavors for strict JSON, strict JSON5, and relaxed/extended JSON5 parsing modes.

• The Printer abstraction allows to consume a stream of tokens directly as provided by the Scanner back into a identical output byte stream providing a valid JSON, JSON5, or relaxed/extended JSON5 document with proper indentation and comments.

• The Decoder abstraction allows to directly decode the stream of tokens from the Scanner into Go objects using reflection. The Decoder comes in two flavors supporting a native and a precise type decoding.

• The Encoder abstraction allows to encode Go objects into a stream of tokens that can be consumed by the Printer to produce a valid JSON, JSON5, or relaxed/extended JSON5 document.

• The Filter abstraction allows to dynamically filter a stream of tokens provided by the Scanner according to a specified filter function. The default filters allow to skip comments, whitespace, and other tokens that are not matching a specific JSON, JSON5, or relaxed/extended JSON5 document standard.

• The Parser abstraction allows to validate the stream of tokens provided by the Scanner and - if requested - to build an abstract syntax tree, that can be used for analysis and processing.

Future plans

The following features are planned for the future, but not yet implemented:

• Split the Reader, Scanner, and Decoder into separate packages, so that they can be used independently and reused in other projects, and advance tests to public interface testing.

• Create different Scanner implementations for strict JSON, strict JSON5, as well as extended and relaxed JSON5 parsing modes - with and without tracking of character and line numbers. For strict JSON and JSON5 parsing modes, the scanner needs to define a clear memory and failure model.

  *Note: The relaxed and extended JSON5 parsing mode is absolutely forgiving and never fails, but may produce invalid token series for the Decoder.

• Create a non-releasing Reader, that allows to access the underlying data without releasing the buffer to enable permanent zero-copy decoding of the data.

• Create specialized Decoder implementations for different parsing modes and different default value types, i.e. big.Int and big.Float vs int64 and float64.

• Create an extended and relaxed JSON5 Emitter supporting exactly the token parsing events of the relaxed and extended JSON5 Scanner to output the unchanged or patched JSON data without ever creating a decoded object.

• Create a Parser that can parse JSON5 data into an abstract syntax tree, allowing for more advanced manipulation and analysis of the JSON5 structure.

• Create an Encoder that can encode Go objects into minimal relaxed JSON5 data, that can be decoded by the Decoder without loss of any information.

• Create JSONPatch support to allow for efficient, on-the-fly patching of JSON data while scanning, parsing, or decoding it.

Open questions:

• Should we eliminate defensive error handling in the Decoder that can not happen due to the Scanner implementation and just panic in these cases?

Building

This project is using a custom build system called go-make, that provides default targets for most common tasks. Makefile rules are generated based on the project structure and files for common tasks, to initialize, build, test, and run the components in this repository.

To get started, run one of the following commands.

make help
make show-targets

Read the go-make manual for more information about targets and configuration options.

Not: go-make installs pre-commit and commit-msg hooks calling make commit to enforce successful testing and linting and make git-verify message to validate whether the commit message is following the conventional commit best practice.

Terms of usage

This software is open source under the MIT license. You can use it without restrictions and liabilities. Please give it a star, so that I know. If the project has more than 25 Stars, I will introduce semantic versions v1.

Contributing

If you like to contribute, please create an issue and/or pull request with a proper description of your proposal or contribution. I will review it and provide feedback on it as fast as possible.

Disclaimer

This software is developed with the help of AI following the highest human standards. All actions executed by AI are carefully reviewed, counter-checked, and corrected with the highest human standards and quality goals in mind. No AI generate code is allowed to be merged or released without a careful human reviews to prevent systematic degeneration of coding standards and code quality.

Acknowledgements

This work is inspired by the great work of Dave Chaney pkg/json and his article about Building a high-performance JSON parser.