Release v1.2.2: The Streaming Update & Extreme Chunking π
This major release introduces the StreamDecoder β a highly optimized suite of tools for parsing infinite JSON arrays and massive network streams without loading the entire payload into memory. It also brings unprecedented raw extraction speeds using a brand-new dedicated AVX2 assembly routine.
π Key New Features
-
StreamDecoder (Standard
Next)
A low-allocation stream parser that extracts and unmarshals objects sequentially from anyio.Reader. Perfect for parsing multi-gigabyte files with minimal RAM footprint (~41MB) at speeds exceeding 470 MB/s. -
Zero-Allocation Async Streaming (
NextChan)
Need to process objects on the main thread while the parser runs in the background?NextChanprovides asynchronous Producer-Consumer streaming. Unlike other libraries that cause massive GC pressure or data races, SilentJSON uses an internal Ring Buffer, achieving 0 GC allocations during streaming at ~378 MB/s. -
Extreme Stream Chunking (
NextRawBlock)
When you just need to slice a massive JSON array into smaller raw byte chunks (e.g., to proxy to a database or save to disk) without unmarshaling to Go structs. Powered by the new dedicatedfindObjectBoundariesEarlyExitASMroutine, it scans and slices streams at an incredible ~4.1 GB/s with zero allocations!
π Improvements & Optimizations
- Dedicated Early-Exit Assembly: Isolated the streaming assembly logic into
findObjectBoundariesEarlyExitASMto ensure the core parser (UnmarshalArrayParallel) maintains its peak throughput (~3.3 GB/s) without any branching overhead. silent-chunkerExample: Added a fully runnable, real-world utility insamples/silent-chunker/demonstrating how to useNextRawBlockto rapidly slice massive JSON files into 10MB chunks and save them to disk.- Documentation & Benchmarks: Completely updated
README.mdwith new Mermaid.js charts, streaming benchmarks, and comprehensiveNextRawBlockusage examples.
π Performance Summary (Streaming)
| Feature | Throughput | Allocations |
|---|---|---|
NextRawBlock (Raw chunk extraction) |
~4150 MB/s | 0 |
Next (Full Go Struct Binding) |
~470 MB/s | Zero-alloc iteration |
NextChan (Async Channel Ring Buffer) |
~378 MB/s | Zero-alloc iteration |
(Note: Standard streaming disables zero-copy strings for memory safety when reading from io.Reader buffers).
Upgrade today to start processing infinite JSON arrays faster than ever!