Skip to content

PyIngestKit v0.4.0 - Dataset Diff, Replay & Versioning

Choose a tag to compare

View all tags
@tawounfouet tawounfouet released this 04 Sep 19:11
· 198 commits to main since this release
v0.4.0
b42e0c9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

PyIngestKit v0.4.0 — Diff, Replay & Versioning Release

Summary

PyIngestKit v0.4.0 delivers the complete Diff, Versioning, Snapshots, and Replay surface while strictly maintaining backward compatibility with V0.1-V0.3 reference jobs and core contracts.

Highlights

  • Dataset Fingerprinting & Diff Engine: Row hashing, canonical type serialization, keyed/keyless row and schema diff calculation (DatasetDiffer, DatasetDiff).
  • Diff Reports & Runtime Observation: Automated JSON/Markdown diff report generation, event-bus integration (DIFF_STARTED, DIFF_COMPLETED, DIFF_REPORT_WRITTEN), and DiffRecord metadata persistence.
  • Dataset Versioning & Snapshots: Content-addressed dataset versions (DatasetVersionStore), JSON snapshots (SnapshotCodec), atomic PublishedDataset pointers, CLI pyingest versions and pyingest published.
  • Deterministic RAW Replay & Lineage: Strict historical RAW artifact replay (ReplayService), CLI pyingest replay <run_id>, lineage tracking, and parameters reproducibility.
  • 7 Reference Jobs: Includes demo.versioned_ndjson demonstrating offline V1 -> V2 -> diff -> publish -> strict RAW replay.

Governance

  • Tag v0.4.0 is permanently immutable. Any subsequent patch will be published as v0.4.1.
Assets 2
Loading