Skip to content

Parity v0.1.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@TFMV TFMV released this 20 Mar 10:39
· 18 commits to main since this release
v0.1.0
b1cc92e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Parity

Parity is a high-performance dataset comparison tool designed to efficiently detect and report differences between large datasets. By leveraging the power of Apache Arrow for in-memory columnar data processing, Parity can handle massive datasets with speed and efficiency.

Features

  • High-Speed Dataset Diffing: Compare large datasets efficiently using vectorized, batch-wise operations
  • Multiple Data Sources: Support for Arrow IPC, Parquet, CSV files, and ADBC-compatible databases
  • Comprehensive Diff Reports: Identify added, deleted, and modified records with column-level detail
  • Arrow-Powered Analysis: Leverage Arrow's in-memory columnar format for high-performance operations
  • Streaming Processing: Handle multi-terabyte datasets without loading them entirely in memory
  • Parallel Execution: Utilize Go's concurrency model for processing partitions simultaneously
  • Flexible Output: Export results in various formats including Arrow IPC, Parquet, JSON, Markdown, and HTML
Assets 2
Loading