jsonq

Developer documentation for contributors.

User documentation: See DOCS.md

Overview

jsonq is a lightweight JSON query engine that implements a custom expression language for querying and transforming JSON data. The library provides a functional programming interface with method chaining, similar to JSONPath but with richer expression support.

Architecture

Core Components

Tokenizer (tokenizePipeline)

Parses query strings into token streams
Handles root reference ($), property access (.prop), array indexing ([0]), and method calls (.method())
Tracks token positions for extracting method parameters

Evaluator (evaluatePipeline)

Executes token pipeline against data
Maintains current value through transformations
Handles nested data navigation

Expression Parser (parseExpr)

Recursive descent parser for expressions
Generates AST with nodes: literal, variable, property, arrayIndex, unary, binary, methodCall
Supports operators: arithmetic, comparison, logical

Expression Evaluator (evalExpr, evalNode)

Detects arrow function syntax with regex
Creates variable bindings for named parameters
Evaluates AST nodes with context (data, item, index, bindings)

Method Executor (applyMethod)

Switch statement dispatching to method implementations
Propagates bindings through nested method calls
Handles recursive evaluation for nested transformations

Public API

export { query };

query(expression, data)

Main entry point
Returns transformed data based on expression

Development

Setup

npm install
npm test

Running Tests

npm test

Test suite includes 98 tests covering:

Tokenization and parsing
All methods (map, filter, sort, etc.)
Expression evaluation (arithmetic, logical, comparison)
Nested method calls
Arrow function syntax and underscore shorthand
Root ($) and array ($array) variable access
Edge cases and error handling

Building

npm run build

Outputs to dist/:

index.cjs - CommonJS
index.mjs - ES Module
index.umd.js - UMD (browser)
index.d.ts - TypeScript definitions

Implementation Details

Supported Methods

const METHODS = [
  "map",
  "filter",
  "join",
  "sort",
  "unique",
  "first",
  "last",
  "count",
  "sum",
  "avg",
  "min",
  "max",
];

Token Types

root - $ reference
property - .prop access
arrayIndex - [0] access
method - .method() call with params

Expression Syntax Features

Variable syntaxes:

Root reference: $
Built-in variables: $item, $index, $array
Underscore shorthand: _ (alias for $item)
Arrow functions: x => x.prop (creates binding for x)

Arrow function detection:

const arrowMatch = expr.match(/^([a-zA-Z_][a-zA-Z0-9_]*)\s*=>\s*(.+)$/);

Binding propagation: Arrow function bindings are passed through nested method calls, enabling parent scope access in nested iterations.

AST Node Types

literal - Numbers, strings, booleans, null, undefined
variable - $, $item, $index, $array, _, or named params
property - Property access (obj.prop)
arrayIndex - Array indexing (arr[0])
unary - Unary operators (!, -)
binary - Binary operators (arithmetic, comparison, logical)
methodCall - Nested method invocations

Sort Implementation

Descending sort detected by - prefix:

const descending = expr.startsWith("-");
const sortExpr = descending ? expr.substring(1) : expr;

Date Handling

Dates prefixed with @ are parsed to timestamps:

if (token.startsWith("@")) {
  return new Date(token.substring(1)).getTime();
}

Contributing

Code Style

No comments in implementation code
Variable names provide documentation
Concise, functional style
Manual parsing (no external parser libraries)

Adding Methods

Add method name to METHODS array
Add case to applyMethod switch statement
Propagate bindings parameter for nested support
Add tests to index.test.js
Update DOCS.md with user-facing documentation

Adding Operators

Update tokenizeExpr regex to include new operator
Add operator precedence in parseExpr
Handle in evalNode binary operator cases
Add tests

Testing Guidelines

Test each method with basic usage
Test chaining combinations
Test edge cases (empty arrays, null values)
Test error conditions
Test nested method scenarios
Test all three syntax styles ($item, _, arrow functions)

Project Structure

├── src/
│   ├── index.js         # Main implementation
│   └── cli.js           # CLI tool (if applicable)
├── test/
│   └── index.test.js    # Test suite
├── examples/
│   └── example.js       # Usage examples
├── dist/                # Build outputs (generated)
├── DOCS.md              # User documentation
├── README.md            # Developer documentation (this file)
├── LICENSE              # MIT License
├── .gitignore
├── package.json
├── rollup.config.mjs
└── jest.config.js

Design Decisions

Manual Parsing

We use a manual tokenizer/parser instead of a library to:

Keep bundle size minimal
Avoid external dependencies
Maintain full control over syntax
Optimize for our specific use case

Three Syntax Styles

Supporting $item, _, and arrow functions provides:

Backward compatibility ($item)
Conciseness (_)
Clarity and parent scope access (arrow functions)

No Pipeline Operator

The > operator was removed to avoid:

Confusion with greater-than comparisons
Additional parsing complexity
Preference for dot-chaining which is more familiar

Method Chaining Only

All operations chain with .method() syntax for consistency with JavaScript's native array methods.

Performance Considerations

No AST caching (expressions evaluated fresh each time)
Recursive descent parser (adequate for expression complexity)
Methods execute eagerly (no lazy evaluation)
Suitable for small to medium datasets

Known Limitations

No async support
No custom function definitions
Single-parameter arrow functions only
No array/object literal construction in expressions
Limited to JavaScript-compatible JSON data

Release Process

Create a branch with the appropriate naming convention (see below)
Make your changes
Run tests: npm test
Build: npm run build
Commit changes
Create a pull request to main
Once merged, the GitHub Actions workflow will automatically:
- Bump the version based on branch name
- Build the package
- Publish to npm with provenance

Branch Naming Convention

The automated release workflow determines the version bump based on your branch name:

Patch release (1.0.0 → 1.0.1): Use fix/* or patch/*
- Example: fix/cli-error-handling, patch/typo-in-docs
Minor release (1.0.0 → 1.1.0): Use add/* or minor/*
- Example: add/new-filter-method, minor/improve-performance
Major release (1.0.0 → 2.0.0): Use vnext/* or major/*
- Example: vnext/breaking-api-change, major/remove-deprecated-methods

If your branch name doesn't match any pattern, it defaults to a major version bump.

License

MIT query("$.users.map($item.name)", data);

// Arithmetic query("$.items.map(price * 1.1)", data); query("$.items.map(_.price * 1.1)", data);

// Complex expressions query("$.users.map(u => u.age * 2 + 10)", data);


### filter(condition)

Select elements matching a condition.

```javascript
// Comparison operators
query("$.users.filter(age > 25)", data);
query('$.users.filter(name == "Alice")', data);

// Using underscore
query("$.users.filter(_.age > 25)", data);

// Using arrow syntax
query("$.users.filter(u => u.age > 25 && u.active)", data);

// Logical operators
query("$.users.filter(age > 25 && active == true)", data);
query("$.users.filter(age < 20 || age > 60)", data);

// Negation
query("$.users.filter(!active)", data);
query("$.users.filter(!_.deleted)", data);

sort(expression)

Sort array elements. Use negative expression for descending order.

// Sort ascending (default)
query("$.users.sort(age)", data);
query("$.users.sort(name)", data);

// Sort descending (use negative)
query("$.users.sort(-age)", data);
query("$.users.sort(-name)", data);

// Sort with expressions
query("$.items.sort(price * quantity)", data);
query("$.items.sort(-(price * quantity))", data); // Descending

join(separator)

Join array elements into a string.

query('$.tags.join(", ")', data);
query('$.users.map(name).join(" | ")', data);

unique(expression)

Get unique values from an array.

// Unique primitives
query("$.tags.unique($item)", data);

// Unique by property
query("$.users.unique(department)", data);

first()

Get the first element.

query("$.users.first()", data);
query("$.users.filter(active).first()", data);

last()

Get the last element.

query("$.users.last()", data);
query("$.users.sort(age).last()", data);

count()

Count array elements.

query("$.users.count()", data);
query("$.users.filter(active).count()", data);

sum(expression)

Sum values in an array.

query("$.items.sum(price)", data);
query("$.numbers.sum($item)", data);

avg(expression)

Calculate average of values.

query("$.users.avg(age)", data);
query("$.items.avg(price * quantity)", data);

min(expression)

Find minimum value.

query("$.users.min(age)", data);
query("$.items.min(price)", data);

max(expression)

Find maximum value.

query("$.users.max(age)", data);
query("$.items.max(price)", data);

Expressions

Variables

Built-in variables:

$item - Current element in iteration
$index - Current index in iteration
$data - Root data object
_ - Shorthand for $item (concise syntax)

Arrow function syntax: Use arrow functions to name your parameters:

// Named parameter (any name you want)
query("$.users.map(user => user.name)", data);
query("$.users.filter(person => person.age > 25)", data);

// Access parent scope in nested iterations
query(
  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
  data,
);

Examples:

// Using built-in variables
query("$.users.map($item.name)", data);
query("$.users.map($index)", data);
query("$.users.filter($item.age > $data.minAge)", data);

// Using underscore shorthand (more concise)
query("$.users.map(_.name)", data);
query("$.items.filter(_.price < 100)", data);

// Using arrow syntax for clarity
query("$.users.map(u => u.name)", data);
query("$.items.filter(item => item.price < 100)", data);

Nested references:

// Underscore shadows in nested contexts
query("$.departments.map(_.employees.map(_.name))", data);
// Outer _ = department, inner _ = employee (shadowed)

// Use arrow syntax when you need parent access
query(
  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
  data,
);

// Mix both styles
query("$.groups.map(g => g.items.filter(_.value > 10).map(_.name))", data);

Operators

Arithmetic: +, -, *, /, %

query("$.items.map(price * 1.2)", data);
query("$.users.filter(age % 2 == 0)", data);

Comparison: ==, !=, <, >, <=, >=

query("$.users.filter(age >= 18)", data);
query("$.items.filter(stock != 0)", data);

Logical: &&, ||, !

query("$.users.filter(age > 18 && active == true)", data);
query("$.items.filter(inStock || onOrder)", data);
query("$.users.filter(!deleted)", data);

Literals

Numbers: 42, 3.14, -10, 1.5e10
Strings: "hello", 'world'
Booleans: true, false
Null: null
Undefined: undefined

query("$.users.filter(age > 25)", data);
query('$.users.filter(name == "Alice")', data);
query("$.users.filter(deleted == null)", data);

Dates

Use @ prefix for ISO date strings:

const data = [
  { event: "Launch", date: "2024-01-15" },
  { event: "Update", date: "2024-06-20" },
];

query("filter(date > @2024-03-01).map(event)", data);
// => ['Update']

// With timestamps
query("filter(createdAt > @2024-01-01T12:00:00Z)", data);

Property Access

Access nested properties in expressions:

query("$.users.map($item.address.city)", data);
query("$.users.map(_.address.city)", data);
query("$.users.map(u => u.address.city)", data);
query("$.items.filter($item.specs.weight > 100)", data);

Array Access

Use brackets in expressions:

query("$.users.map($item.tags[0])", data);
query("$.users.map(_.tags[0])", data);
query("$.data.map($item.values[$index])", data);

Nested Method Calls

Call methods within expressions for complex transformations:

const data = {
  departments: [
    {
      name: "Engineering",
      employees: [
        { name: "Alice", age: 30 },
        { name: "Bob", age: 25 },
        { name: "Carol", age: 35 },
      ],
    },
    {
      name: "Sales",
      employees: [
        { name: "Dave", age: 40 },
        { name: "Eve", age: 28 },
      ],
    },
  ],
};

// Filter and map employees within each department (concise)
query("$.departments.map(_.employees.filter(age > 28).map(name))", data);
// => [['Alice', 'Carol'], ['Dave']]

// With arrow syntax for parent access
query(
  '$.departments.map(dept => dept.employees.map(emp => dept.name + ": " + emp.name))',
  data,
);
// => [['Engineering: Alice', 'Engineering: Bob', 'Engineering: Carol'], ['Sales: Dave', 'Sales: Eve']]

// Count employees per department
query("$.departments.map(_.employees.count())", data);
// => [3, 2]

// Average age per department
query("$.departments.map(_.employees.avg(age))", data);
// => [30, 34]

// Multiple levels of nesting with underscore
query("$.departments.map(_.employees.sort(age).first().name)", data);
// => ['Bob', 'Eve']

Examples

E-commerce

const orders = {
  items: [
    { name: "Laptop", price: 999, quantity: 2, inStock: true },
    { name: "Mouse", price: 25, quantity: 5, inStock: true },
    { name: "Keyboard", price: 75, quantity: 0, inStock: false },
  ],
};

// Total value of in-stock items
query("$.items.filter(inStock).map(price * quantity).sum($item)", orders);
// => 2048

// Most expensive item name
query("$.items.sort(price).last().name", orders);
// => 'Laptop'

// Average price
query("$.items.avg(price)", orders);
// => 366.33...

User Management

const company = {
  departments: [
    {
      name: "Engineering",
      employees: [
        { name: "Alice", salary: 120000, startDate: "2020-01-15" },
        { name: "Bob", salary: 95000, startDate: "2021-06-01" },
      ],
    },
    {
      name: "Sales",
      employees: [{ name: "Carol", salary: 85000, startDate: "2019-03-10" }],
    },
  ],
};

// All department names
query('$.departments.map(name).join(", ")', company);
// => 'Engineering, Sales'

// High earners in each department
query(
  "$.departments.map($item.employees.filter(salary > 100000).map(name))",
  company,
);
// => [['Alice'], []]

// Recent hires
query(
  "$.departments[0].employees.filter(startDate > @2021-01-01).map(name)",
  company,
);
// => ['Bob']

// Department sizes
query("$.departments.map($item.employees.count())", company);
// => [2, 1]

Data Analysis

const metrics = {
  daily: [
    { date: "2024-01-01", views: 1200, clicks: 45 },
    { date: "2024-01-02", views: 1500, clicks: 67 },
    { date: "2024-01-03", views: 980, clicks: 32 },
  ],
};

// Click-through rates
query("$.daily.map(clicks / views * 100)", metrics);
// => [3.75, 4.47, 3.27]

// Best performing day
query("$.daily.sort(clicks).last().date", metrics);
// => '2024-01-02'

// Total engagement
query("$.daily.sum(views + clicks)", metrics);
// => 3824

API Reference

query(expression, data)

Execute a query against data.

Parameters:

expression (string): Query expression
data (any): Data to query

Returns: Query result

const result = query("$.users.filter(age > 25)", data);

tokenize(expression)

Parse expression into tokens (exported for testing/debugging).

Parameters:

expression (string): Query expression

Returns: Array of tokens

import { tokenize } from "jsonq";
const tokens = tokenize("$.users.map(name)");

evaluate(tokens, data)

Evaluate parsed tokens against data (exported for testing/debugging).

Parameters:

tokens (Array): Parsed tokens
data (any): Data to evaluate

Returns: Evaluation result

import { evaluate, tokenize } from "jsonq";
const tokens = tokenize("$.users.count()");
const result = evaluate(tokens, data);

Advanced Usage

Complex Filters

Combine multiple conditions:

query(
  `
  $.users
    .filter(age >= 21 && age <= 65)
    .filter(active == true)
    .filter(verified == true)
    .sort(name)
`,
  data,
);

Nested Access Patterns

Access deeply nested data:

query('$.company.departments[0].teams[2].members.filter(role == "lead")', data);

Combine nested methods with property navigation:

// Get top performer from each department
query("$.departments.map($item.employees.sort(performance).last())", data);

// Filter departments by employee criteria
query(
  "$.departments.filter($item.employees.filter(certified == true).count() > 5)",
  data,
);

// Aggregate across multiple levels
query("$.regions.map($item.stores.map($item.sales.sum(amount)))", data);

Dynamic Calculations

Use expressions in any method:

query("$.products.filter((price - cost) / price > 0.3)", data);

Index-Based Operations

Use $index for position-dependent logic:

query("$.items.filter($index % 2 == 0)", data); // Even indices
query("$.items.map($index * 10 + $item.value)", data);

Type Handling

Null and Undefined

Safe navigation with optional chaining behavior:

const data = [{ user: { name: "Alice" } }, { user: null }, {}];

query("map($item.user.name)", data);
// => ['Alice', undefined, undefined]

Missing Properties

Accessing non-existent properties returns undefined:

query("$.users.map(nonexistent)", data);
// => [undefined, undefined, ...]

Type Coercion

Dates are automatically coerced for comparison:

query("filter(dateString > @2024-01-01)", data);

Performance Tips

Filter Early: Apply filters before expensive operations

// Good
query("$.users.filter(active).map(expensiveOperation)", data);

// Less efficient
query("$.users.map(expensiveOperation).filter(active)", data);

Avoid Redundant Sorts: Sort only when necessary

// If you only need first/last, consider avoiding sort
query("$.users.sort(age).first()", data);

Use Specific Expressions: More specific filters run faster

// More specific
query("$.users.filter(id == 123)", data);

// Less specific
query("$.users.filter(id > 0)", data);

Error Handling

The library throws errors for:

Unknown methods
Invalid expressions
Unknown variables
Unexpected tokens
Invalid literals

try {
  query("$.users.unknownMethod()", data);
} catch (error) {
  console.error(error.message); // "Unknown method: unknownMethod"
}

Testing

Run the test suite:

npm test

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 26 Commits
.github/workflows		.github/workflows
examples		examples
src		src
test		test
.eslintrc.json		.eslintrc.json
.gitattributes		.gitattributes
.gitignore		.gitignore
DOCS.md		DOCS.md
LICENSE		LICENSE
README.md		README.md
jest.config.js		jest.config.js
package-lock.json		package-lock.json
package.json		package.json
rollup.config.mjs		rollup.config.mjs

Folders and files

Latest commit

History

Repository files navigation

jsonq

Overview

Architecture

Core Components

Public API

Development

Setup

Running Tests

Building

Implementation Details

Supported Methods

Token Types

Expression Syntax Features

AST Node Types

Sort Implementation

Date Handling

Contributing

Code Style

Adding Methods

Adding Operators

Testing Guidelines

Project Structure

Design Decisions

Manual Parsing

Three Syntax Styles

No Pipeline Operator

Method Chaining Only

Performance Considerations

Known Limitations

Release Process

Branch Naming Convention

License

sort(expression)

join(separator)

unique(expression)

first()

last()

count()

sum(expression)

avg(expression)

min(expression)

max(expression)

Expressions

Variables

Operators

Literals

Dates

Property Access

Array Access

Nested Method Calls

Examples

E-commerce

User Management

Data Analysis

API Reference

query(expression, data)

tokenize(expression)

evaluate(tokens, data)

Advanced Usage

Complex Filters

Nested Access Patterns

Dynamic Calculations

Index-Based Operations

Type Handling

Null and Undefined

Missing Properties

Type Coercion

Performance Tips

Error Handling

Testing

License

About

Resources

License

Uh oh!

Stars

Packages