Skip to content

Release v1.2.2: The Streaming Update & Extreme Chunking πŸš€

Choose a tag to compare

@GenshIv GenshIv released this 28 Jun 22:29
6dd044f

This major release introduces the StreamDecoder β€” a highly optimized suite of tools for parsing infinite JSON arrays and massive network streams without loading the entire payload into memory. It also brings unprecedented raw extraction speeds using a brand-new dedicated AVX2 assembly routine.

🌟 Key New Features

  • StreamDecoder (Standard Next)
    A low-allocation stream parser that extracts and unmarshals objects sequentially from any io.Reader. Perfect for parsing multi-gigabyte files with minimal RAM footprint (~41MB) at speeds exceeding 470 MB/s.

  • Zero-Allocation Async Streaming (NextChan)
    Need to process objects on the main thread while the parser runs in the background? NextChan provides asynchronous Producer-Consumer streaming. Unlike other libraries that cause massive GC pressure or data races, SilentJSON uses an internal Ring Buffer, achieving 0 GC allocations during streaming at ~378 MB/s.

  • Extreme Stream Chunking (NextRawBlock)
    When you just need to slice a massive JSON array into smaller raw byte chunks (e.g., to proxy to a database or save to disk) without unmarshaling to Go structs. Powered by the new dedicated findObjectBoundariesEarlyExitASM routine, it scans and slices streams at an incredible ~4.1 GB/s with zero allocations!

πŸ›  Improvements & Optimizations

  • Dedicated Early-Exit Assembly: Isolated the streaming assembly logic into findObjectBoundariesEarlyExitASM to ensure the core parser (UnmarshalArrayParallel) maintains its peak throughput (~3.3 GB/s) without any branching overhead.
  • silent-chunker Example: Added a fully runnable, real-world utility in samples/silent-chunker/ demonstrating how to use NextRawBlock to rapidly slice massive JSON files into 10MB chunks and save them to disk.
  • Documentation & Benchmarks: Completely updated README.md with new Mermaid.js charts, streaming benchmarks, and comprehensive NextRawBlock usage examples.

πŸ“Š Performance Summary (Streaming)

Feature Throughput Allocations
NextRawBlock (Raw chunk extraction) ~4150 MB/s 0
Next (Full Go Struct Binding) ~470 MB/s Zero-alloc iteration
NextChan (Async Channel Ring Buffer) ~378 MB/s Zero-alloc iteration

(Note: Standard streaming disables zero-copy strings for memory safety when reading from io.Reader buffers).


Upgrade today to start processing infinite JSON arrays faster than ever!