Repository navigation
Releases: litepacks/rowpipe
Release list
v2.10.5
[2.10.5] - 2026-09-27
🚀 New Formats & Protocol Adapters
-
Apache Avro (.avro) Support:
-
Added src/readers/avro.ts and src/writers/avro.ts.
-
Supports automatic Avro schema derivation from tabular streams, nested/primitive type mapping, and block-based binary streaming.
-
-
Apache Arrow & Feather (.arrow, .feather) Support:
-
Added src/readers/arrow.ts and src/writers/arrow.ts.
-
High-performance vectorized RecordBatch streaming, binary serialization, and zero-copy columnar conversions via apache-arrow.
-
-
XML (.xml) Support:
-
Added src/readers/xml.ts and src/writers/xml.ts.
-
Automatic detection of repeating row elements in hierarchical XML structures, automatic primitive coercion, and streaming XML serialization.
-
-
NDJSON (.ndjson) Support:
- Full native support for newline-delimited JSON streams in src/readers/jsonl.ts.
-
Magic-Byte Sniffing (Automatic Format Detection):
- Added src/readers/sniff.ts with sniffFormatFromBuffer and sniffFormatFromFile to identify file formats (Parquet, Arrow, Avro, XML, JSON, CSV) from header magic bytes even when extensions are missing.
⚡ Performance & Zero-Allocation Pipeline
-
Top-K Sorting (src/transforms/top.ts):
- Min/Max-Heap candidate pre-filtering and single-field direct comparison fast-path (3.3x speedup, 500k rows: 1.1s → 328ms).
-
JSON & JSONL Writers (src/writers/jsonl.ts, src/writers/json.ts):
- Native V8 C++ string buffer aggregation and memory allocation elimination (2.4x throughput, 188 MB/s → 451 MB/s).
-
Schema Inference & Profiler (src/analytics/schema-inference.ts, src/analytics/semantic-types.ts):
- Replaced inner Object.entries loops with for..in and added regex length guards (3.0x speedup, 200k rows: 1.83s → 596ms).
-
Group-By & Reduce (src/analytics/reduce.ts):
- Added single-grouping-key direct string map fast-path and eliminated intermediate wrapper objects (3.2x speedup, 200k rows: 735ms → 228ms).
-
Spillable KeyStore & Unique (src/utils/keystore.ts, src/transforms/unique.ts):
- Added synchronous fast-path (hasSync, addSync) bypassing ~400,000 Promise instantiations during in-memory streaming (2.5x speedup, 200k rows: 840ms → 334ms).
-
Hash Join (src/transforms/join/join.ts, src/transforms/join/index-storage.ts):
- Zero-allocation createRowMerger using compiled Set/Map lookups (eliminated 2.4M array allocations per 200k rows) and synchronous index probes (2.4x speedup, 422,000+ rows/sec).
🛠️ Developer Experience (DX) & CLI
-
rowpipe formats Command (src/cli/commands/formats.ts):
- Displays all 10 supported data formats, extensions, streaming status, and read/write capabilities in tabular or JSON formats.
-
Shell Auto-Completion (src/cli/commands/completion.ts):
- Introduced rowpipe completion --install supporting Bash, Zsh, and Fish shell completions.
-
Smart Command Fuzzy Matcher (src/utils/fuzzy.ts):
- Provides "Did you mean?" suggestions on mistyped commands or arguments.
-
Dry-Run & Fluent Pipeline API (src/core/pipeline.ts):
-
Added --dry-run flag to validate transformations and print planned execution stages without running full I/O.
-
Fluent method chaining on pipeline instances (createPipeline().pipe(...)).
-
🤖 Model Context Protocol (MCP) Server
-
Added rowpipe_formats MCP tool exposing the runtime capabilities matrix to AI agents.
-
Full integration of Avro, Arrow, and XML formats into MCP tools: rowpipe_inspect, rowpipe_query, rowpipe_schema, rowpipe_stats, rowpipe_sample, rowpipe_convert, rowpipe_diff, rowpipe_profile, and rowpipe_table.
🧪 Test Suite & Reliability
-
600+ new test assertions covering Arrow, Avro, XML, NDJSON, dry-run pipelines, fuzzy matching, and shell completion.
-
Full suite passing: 35 test files, 399 tests passing (100% green).
Full Changelog: v2.10.4...v2.10.5
Full Changelog: v2.10.4...v2.10.5
v2.10.4
v2.10.3
v2.10.2
Full Changelog: v2.10.1...v2.10.2
Full Changelog: v2.10.1...v2.10.2
Full Changelog: v2.10.1...v2.10.2
v2.10.1
Full Changelog: https://github.com/litepacks/rowpipe/commits/v2.10.1