JSONParser - High-Performance JSON Parser for Go

Overview

JSONParser is an enhanced fork of the popular buger/jsonparser library with significant improvements in streaming capabilities, memory management, and performance optimizations. While maintaining the original's excellent API design, this version adds powerful new features for production use cases.

Why JSONParser?

The Problem

Standard JSON unmarshaling in Go (encoding/json) requires:

• Defining structs that match your JSON structure
• Allocating memory for the entire decoded structure
• Processing the entire JSON document even if you only need a few fields

This becomes problematic when:

• Working with large JSON documents
• You only need a few fields from complex nested structures
• Performance and memory usage are critical
• JSON structure is dynamic or unknown at compile time

The Solution

JSONParser provides:

• Direct field access - Get any field by path without unmarshaling
• Zero allocations for most operations - Uses stack-allocated buffers
• Streaming support - Process large JSON files without loading them entirely into memory
• No schema required - Works with any valid JSON structure
• Type-safe operations - Returns proper Go types with error handling

Features

• 🚀 High Performance - Up to 10x faster than standard library for selective field access
• 💾 Memory Efficient - Stack-allocated buffers, minimal heap allocations
• 🔍 Path-based Access - Navigate nested structures with simple key paths
• 🌊 Streaming API - Process large files incrementally
• 🛠️ Rich API - Support for objects, arrays, primitives, and complex navigation
• ✅ Type Safe - Strong typing with proper error handling
• 🔄 Mutation Support - Modify JSON documents in place

Installation

go get github.com/SergeiSkv/jsonparser

Quick Start

package main

import (
    "fmt"
    "github.com/SergeiSkv/jsonparser"
)

func main() {
    data := []byte(`{
        "user": {
            "name": "John Doe",
            "age": 30,
            "emails": ["john@example.com", "doe@example.com"]
        }
    }`)

    // Get nested string value
    name, err := jsonparser.GetString(data, "user", "name")
    if err != nil {
        panic(err)
    }
    fmt.Println("Name:", name) // Output: John Doe

    // Get nested number
    age, err := jsonparser.GetInt(data, "user", "age")
    if err != nil {
        panic(err)
    }
    fmt.Println("Age:", age) // Output: 30

    // Iterate over array
    jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
        email, _ := jsonparser.ParseString(value)
        fmt.Println("Email:", email)
    }, "user", "emails")
}

Core Concepts

Value Types

JSONParser recognizes the following value types:

const (
    NotExist = ValueType(iota) // Key path not found
    String                      // JSON string
    Number                      // JSON number
    Object                      // JSON object
    Array                       // JSON array
    Boolean                     // JSON boolean (true/false)
    Null                        // JSON null
    Unknown                     // Unknown type
)

Path Navigation

Access nested values using variadic string arguments:

// Access data.user.profile.avatar
avatar, _, _, err := jsonparser.Get(data, "user", "profile", "avatar")

// Access array element: data.items[2]
item, _, _, err := jsonparser.Get(data, "items", "[2]")

API Reference

Basic Operations

Get - Universal getter

value, dataType, offset, err := jsonparser.Get(data, keys...)

Returns raw bytes, type information, and offset in the original data.

Type-specific Getters

// String
str, err := jsonparser.GetString(data, keys...)

// Numbers
intVal, err := jsonparser.GetInt(data, keys...)
floatVal, err := jsonparser.GetFloat(data, keys...)

// Boolean
boolVal, err := jsonparser.GetBoolean(data, keys...)

// Unsafe string (no allocations, references original bytes)
unsafeStr, err := jsonparser.GetUnsafeString(data, keys...)

Array Operations

ArrayEach - Iterate over array elements

jsonparser.ArrayEach(data, func(value []byte, dataType ValueType, offset int, err error) {
    // Process each element
}, keys...)

Example: Processing an array of objects

data := []byte(`{
    "items": [
        {"id": 1, "name": "Item 1"},
        {"id": 2, "name": "Item 2"}
    ]
}`)

jsonparser.ArrayEach(data, func(value []byte, dataType ValueType, offset int, err error) {
    id, _ := jsonparser.GetInt(value, "id")
    name, _ := jsonparser.GetString(value, "name")
    fmt.Printf("Item %d: %s\n", id, name)
}, "items")

Object Operations

ObjectEach - Iterate over object key-value pairs

jsonparser.ObjectEach(data, func(key []byte, value []byte, dataType ValueType, offset int) error {
    fmt.Printf("Key: %s, Value: %s\n", key, value)
    return nil
}, keys...)

EachKey - Process specific keys efficiently

paths := [][]string{
    {"user", "name"},
    {"user", "email"},
    {"settings", "theme"}
}

jsonparser.EachKey(data, func(idx int, value []byte, vt ValueType, err error) {
    switch idx {
    case 0: // user.name
        fmt.Println("Name:", string(value))
    case 1: // user.email
        fmt.Println("Email:", string(value))
    case 2: // settings.theme
        fmt.Println("Theme:", string(value))
    }
}, paths...)

Modification Operations

Set - Update or add a value

data, err = jsonparser.Set(data, []byte(`"new value"`), "path", "to", "field")

Delete - Remove a field

data = jsonparser.Delete(data, "path", "to", "field")

Streaming API

Process large JSON files without loading them entirely:

decoder := jsonparser.NewStreamDecoder(reader, 
    jsonparser.WithReadBufferSize(8192),
    jsonparser.WithValueBufferSize(4096),
)

for decoder.Next() {
    // Process each top-level element
    key, value, valueType, _, _, err := decoder.ParseObjectKeyPair()
    if err != nil {
        break
    }
    // Process key-value pair
}

Performance Optimizations

Memory Management

1. Buffer Pooling: Reusable buffers for large allocations

// Internal buffer pool implementation reduces GC pressure
var bufferPool = sync.Pool{
    New: func() interface{} {
        buf := make([]byte, 4096)
        return &buf
    },
}

2. Stack Allocation: Small buffers allocated on stack

const unescapeStackBufSize = 64 // Stack-allocated buffer for small strings

3. In-place Operations: Modifications without copying entire document

// removeQuotes operates in-place when possible
func removeQuotes(b []byte) []byte {
    if len(b) >= 2 && b[0] == '"' && b[len(b)-1] == '"' {
        return b[1 : len(b)-1] // No allocation, just reslicing
    }
    // ... fallback for complex cases
}

Parsing Optimizations

1. Direct Byte Comparison: Avoid string conversions

// Instead of bytes.Equal(value, []byte("true"))
if len(value) == 4 && value[0] == 't' && value[1] == 'r' && value[2] == 'u' && value[3] == 'e' {
    return true, nil
}

2. Optimized Integer Parsing: Custom parseInt ~2x faster than strconv

func parseInt(bytes []byte) (int64, bool, bool) {
    // Custom implementation optimized for JSON numbers
    // Handles overflow detection efficiently
}

3. Escape Sequence Handling: Only unescape when necessary

if bytes.IndexByte(value, '\\') == -1 {
    return string(value), nil // No escapes, direct conversion
}
// Unescape only when needed

Advanced Usage

Working with Unknown JSON Structure

var parseJSON func([]byte, string) 
parseJSON = func(data []byte, prefix string) {
    jsonparser.ObjectEach(data, func(key, value []byte, dataType ValueType, offset int) error {
        fullPath := prefix + string(key)
        
        switch dataType {
        case jsonparser.Object:
            parseJSON(value, fullPath + ".")
        case jsonparser.Array:
            fmt.Printf("%s = [array with elements]\n", fullPath)
        default:
            fmt.Printf("%s = %s\n", fullPath, value)
        }
        return nil
    })
}

parseJSON(data, "")

Error Handling

JSONParser provides specific error types for different scenarios:

var (
    ErrKeyPathNotFound       = errors.New("key path not found")
    ErrUnknownValueType      = errors.New("unknown value type")
    ErrMalformedJSON         = errors.New("malformed JSON error")
    ErrMalformedString       = errors.New("value is string, but can't find closing '\"' symbol")
    ErrMalformedArray        = errors.New("value is array, but can't find closing ']' symbol")
    ErrMalformedObject       = errors.New("value looks like object, but can't find closing '}' symbol")
    ErrMalformedValue        = errors.New("value looks like Number/Boolean/None, but can't find its end")
    ErrOverflowInteger       = errors.New("value is number, but overflowed while parsing")
    ErrMalformedStringEscape = errors.New("encountered an invalid escape sequence in a string")
)

Custom Value Processing

// Process values based on their type
value, dataType, _, err := jsonparser.Get(data, "field")
if err != nil {
    return err
}

switch dataType {
case jsonparser.String:
    str, _ := jsonparser.ParseString(value)
    // Process string
case jsonparser.Number:
    num, _ := jsonparser.ParseFloat(value)
    // Process number
case jsonparser.Object:
    // Process nested object
    jsonparser.ObjectEach(value, ...)
case jsonparser.Array:
    // Process array
    jsonparser.ArrayEach(value, ...)
case jsonparser.Boolean:
    bool, _ := jsonparser.ParseBoolean(value)
    // Process boolean
case jsonparser.Null:
    // Handle null
}

Benchmarks

Performance comparison with standard library and other popular JSON libraries:

BenchmarkEncodingJSON-8      100000     15175 ns/op    3272 B/op    69 allocs/op
BenchmarkJSONParser-8       3000000       548 ns/op       0 B/op     0 allocs/op
BenchmarkEasyJSON-8          500000      2717 ns/op     432 B/op     9 allocs/op
BenchmarkFFJSON-8            300000      4321 ns/op     856 B/op    15 allocs/op

// Large payload (10KB JSON)
BenchmarkLargeEncodingJSON-8   5000    287126 ns/op   71737 B/op   351 allocs/op
BenchmarkLargeJSONParser-8   100000     12734 ns/op       0 B/op     0 allocs/op

Use Cases

JSONParser is ideal for:

1. API Gateways - Extract routing information without parsing entire request
2. Log Processing - Extract specific fields from structured logs
3. Data Pipelines - Transform JSON streams efficiently
4. Microservices - Quick validation and field extraction
5. IoT Applications - Process sensor data with minimal memory
6. Configuration Management - Read specific config values
7. Web Scraping - Extract data from JSON APIs

Limitations

• Does not validate entire JSON structure (only the traversed path)
• Modifications return new byte slices (though optimized for performance)
• Not suitable for complex JSON transformations requiring full document structure

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

MIT License - see LICENSE file for details

Comparison with Original buger/jsonparser

This fork extends the original buger/jsonparser with significant enhancements:

New Features Added

1. Advanced Streaming Support

   • StreamDecoder for processing large JSON files incrementally
   • Scanner for low-level token-based parsing
   • Configurable buffer sizes for memory control
   • Callback-based processing for streaming data

2. Enhanced Memory Management

   • Buffer pooling system (pool.go) for reusing allocations
   • Optimized escape handling with in-place operations
   • Improved removeQuotes function that avoids allocations
   • Smart buffer capacity management in ensureCapacity

3. Performance Optimizations

   • Removed recursion in getObjectLength
   • Direct byte comparisons instead of bytes.Equal
   • Optimized stringEnd with better backslash counting
   • Custom parseInt with improved overflow detection
   • Loop optimizations in tokenEnd and nextToken

4. New Configuration Options

   • Config struct with customizable settings
   • Option pattern for flexible initialization
   • Configurable read/value buffer sizes
   • Callback support for custom processing

5. Additional APIs

   • GetType() for quick type detection
   • Stream-specific parsing methods
   • Enhanced error handling with more specific error types
   • Better support for malformed JSON recovery

Core Improvements

Feature	Original buger/jsonparser	This Fork
Streaming	Basic support	Full StreamDecoder with callbacks
Memory Pools	No	Yes, with sync.Pool
Buffer Management	Fixed	Configurable sizes
Escape Handling	Standard	Optimized with stack buffers
Integer Parsing	Standard	~2x faster custom implementation
Recursion	Used in some functions	Eliminated for better performance
Configuration	Minimal	Rich options pattern

Backwards Compatibility

All original APIs are preserved and enhanced:

• Get(), GetString(), GetInt(), etc. - Same signatures, better performance
• ArrayEach(), ObjectEach() - Fully compatible
• Set(), Delete() - Enhanced but compatible
• Error types - Extended but backwards compatible

When to Use This Fork

Choose this fork when you need:

• Streaming large JSON files - StreamDecoder handles files of any size
• Better memory control - Buffer pooling and size configuration
• Higher performance - Optimizations provide 10-30% better performance
• Production robustness - Better error handling and recovery

Stay with the original if:

• You need a minimal dependency
• You're already deeply integrated with the original
• The additional features aren't required for your use case

Acknowledgments

This library is based on the excellent buger/jsonparser by Leonid Bugaev. We've built upon its solid foundation to add streaming capabilities and performance optimizations while maintaining API compatibility.

Additional optimization techniques inspired by:

• simdjson for fast JSON parsing strategies
• fasthttp for zero-allocation techniques
• encoding/json for API design patterns

Support

For bugs and feature requests, please use the GitHub issue tracker.

For questions and discussions, feel free to open a GitHub Discussion.