PyIngestKit v1.0.0 - Stable Framework Contract

PyIngestKit 1.0.0 - Stable Framework Contract

Status: stable (publication is gated on successful post-merge qualification of the exact stable SHA).

PyIngestKit 1.0.0 promotes the fully qualified V1 release candidate to the first protected 1.x
compatibility line. It adds no new provider or orchestration capability at the version transition.

Release lineage

Stable promotion starts from the post-merge-qualified RC1 baseline:

e98a12cc3bbfb634d2fd2257f43049fa1e0333dd

The immutable v0.6.0 release remains the executable historical upgrade baseline.

Stable contents

• framework package version 1.0.0;
• maintained demo package version 1.0.0;
• governed V1 Python public API promoted from stable-candidate to stable 1.x contract;
• explicitly experimental surfaces kept outside the 1.x promise;
• deterministic plugin discovery and failure isolation;
• fail-closed configuration/profile resolution and explicit workspace precedence;
• canonical error/deprecation behavior;
• governed CLI commands, exit-code classes and successful JSON semantics;
• governed logging/redaction behavior;
• logical persistence and portable schema compatibility contracts;
• immutable RAW, deterministic versioning/diff/publication and strict replay;
• PostgreSQL persistence targets and metadata;
• S3-compatible artifact/version storage;
• five representative pilots covering nine maintained reference jobs;
• Python 3.11/3.12/3.13 qualification;
• PostgreSQL 16, MinIO/S3 and cross-host object-storage E2E qualification.

Upgrade evidence

The stable release-check creates real V0.6 versioned run history, versions and publication state from
the exact v0.6.0 tag, upgrades the same isolated environment to the built 1.0.0 wheels and proves:

• historical run state remains readable;
• historical DatasetVersion snapshots remain readable;
• PublishedDataset still points at the historical V2 run;
• strict replay succeeds from historical RAW without live reacquisition;
• expected and actual fingerprints equal the historical published fingerprint.

Stable compatibility promise

All surfaces classified PUBLIC_STABLE_CANDIDATE by the historical A1 inventory and retained through
A2/B1/B2/RC1 are promoted to PUBLIC_STABLE for 1.x. Explicit experimental surfaces remain
experimental.

Stable public paths follow the documented deprecation policy and are retained throughout 1.x unless an
explicitly documented security/correctness emergency requires exceptional action.

Stable distributions

pyingestkit-1.0.0-py3-none-any.whl
pyingestkit-1.0.0.tar.gz
pyingestkit_demo_jobs-1.0.0-py3-none-any.whl
pyingestkit_demo_jobs-1.0.0.tar.gz
SHA256SUMS

CI groups release evidence as:

pyingestkit-v1.0.0-source
pyingestkit-v1.0.0-dist

Release governance

The stable v1.0.0 tag is not created on an unmerged release branch. It is created as an annotated,
immutable tag only after the stable PR is merged with a merge commit and the exact merge SHA passes
post-merge CI and Security. Final release assets and checksums are then published from that qualified
lineage.

The historical v0.6.0 tag and RC1 evidence remain unchanged.

PyIngestKit v0.6.0 - Object Storage

PyIngestKit v0.6.0 - Object Storage

Stable promotion of the qualified V0.6 Object Storage line. No new runtime capability was added after RC1.

Highlights

• portable ArtifactURI / StoredArtifact references;
• immutable remote RAW plus remote reports/manifests through S3ArtifactStore;
• immutable content-addressed remote DatasetVersion snapshots and mutable PublishedDataset pointer through S3DatasetVersionStore;
• MinIO-qualified S3-compatible behavior with boto3 remaining optional;
• strict cross-host replay after deleting the original runner workspace;
• PostgreSQL target idempotency preserved across hosts;
• final public API, S3 config and storage/replay error contracts frozen;
• MinIO CI image pinned by digest and ephemeral random CI credentials;
• Python 3.11/3.12/3.13, PostgreSQL 16, MinIO/S3, full cross-host E2E, release-check, Bandit and pip-audit all green on main.

Qualified main commit: 603d02b0833bc330edc29d00f21ad8d51bfdf0b4
Main CI run: 33963703642
Main Security run: 33963703631

The attached SHA256SUMS covers every attached binary/source artifact.

PyIngestKit v0.5.1 - Release Hygiene

Maintenance release for the V0.5 PostgreSQL Persistence line.

Highlights:

• hardens automatic local .env loading without overriding OS/platform variables;
• prevents implicit parent-directory .env discovery;
• replaces interpolated demo PostgreSQL DDL with SQLAlchemy schema objects;
• proves automatic fixture-only PostgreSQL target-table creation from an absent table;
• fixes the Ruff formatting regression from the V0.5.0 post-release enhancements;
• preserves all V0.5 Target, PostgreSQL COPY, load-mode, idempotency, MetadataStore, versioning and strict replay contracts.

Qualified on Python 3.11, 3.12 and 3.13, PostgreSQL 16, Ruff, Mypy, Bandit, pip-audit, wheel/sdist build and clean-wheel smoke.

v0.5.0 - PostgreSQL Persistence Targets

PyIngestKit V0.5.0 — PostgreSQL Persistence Targets Release

• Promoted fully qualified PostgreSQL persistence release to stable v0.5.0;
• Released PostgresTarget with deterministic schema compatibility, psycopg 3 COPY, transactional APPEND, TRUNCATE_LOAD, and REPLACE;
• Integrated python-dotenv for seamless local environment variable management;
• Enhanced reference jobs with automatic target table preparation and complete PostgreSQL MetadataStore support.

See CHANGELOG.md for full details.

PyIngestKit v0.4.0 - Dataset Diff, Replay & Versioning

PyIngestKit v0.4.0 — Diff, Replay & Versioning Release

Summary

PyIngestKit v0.4.0 delivers the complete Diff, Versioning, Snapshots, and Replay surface while strictly maintaining backward compatibility with V0.1-V0.3 reference jobs and core contracts.

Highlights

• Dataset Fingerprinting & Diff Engine: Row hashing, canonical type serialization, keyed/keyless row and schema diff calculation (DatasetDiffer, DatasetDiff).
• Diff Reports & Runtime Observation: Automated JSON/Markdown diff report generation, event-bus integration (DIFF_STARTED, DIFF_COMPLETED, DIFF_REPORT_WRITTEN), and DiffRecord metadata persistence.
• Dataset Versioning & Snapshots: Content-addressed dataset versions (DatasetVersionStore), JSON snapshots (SnapshotCodec), atomic PublishedDataset pointers, CLI pyingest versions and pyingest published.
• Deterministic RAW Replay & Lineage: Strict historical RAW artifact replay (ReplayService), CLI pyingest replay <run_id>, lineage tracking, and parameters reproducibility.
• 7 Reference Jobs: Includes demo.versioned_ndjson demonstrating offline V1 -> V2 -> diff -> publish -> strict RAW replay.

Governance

• Tag v0.4.0 is permanently immutable. Any subsequent patch will be published as v0.4.1.

v0.3.0 - Quality & Formats Release

Technical Review — PyIngestKit V0.3.0 Quality Gate & Documentation Stabilization

Overview

This technical review documents the corrections made to PyIngestKit V0.3.0 to restore 100% functionality across all quality gates (make quality, make security, make test, make build, make verify, make release-check) and CLI documentation.

---

Identified Issues & Resolutions

1. Security Gate Fix — Elimination of assert in Production Code (Bandit B101)

• Root Cause: bandit -q -r src/pyingestkit examples/plugin_package/src reported two low-severity/high-confidence security issues (B101:assert_used) in src/pyingestkit/contracts/dataset.py at lines 457 and 493:

  minimum = contract.min_length
  assert minimum is not None

  In Python, assert statements can be omitted at bytecode compilation when optimized (python -O), causing critical runtime contract validations to be silently bypassed.
• Fix Applied: Replaced assertions with explicit None checks:

  minimum = contract.min_length
  if minimum is None:
      return True

  maximum = contract.max_length
  if maximum is None:
      return True

• Outcome: bandit scan now reports 0 issues across 5,705 lines of scanned code.

---

2. Quality Gate Fix — Code Formatting & Mypy Type Narrowing

A. Code Formatting (ruff format)

• Root Cause: ruff format --check reported 5 unformatted files across the repository:
  • examples/plugin_package/tests/test_quality_format_jobs.py
  • src/pyingestkit/contracts/dataset.py
  • src/pyingestkit/runtime/runner.py
  • tests/integration/test_quality_reports_runtime.py
  • tests/unit/contracts/test_dataset_contract_v2.py
• Fix Applied: Executed ruff format across src, tests, and examples/plugin_package. All 168 source files are now compliant.

B. Mypy Strict Type Checking (mypy src/pyingestkit)

• Root Cause: Mypy reported an argument type error in src/pyingestkit/contracts/dataset.py:305:

  src/pyingestkit/contracts/dataset.py:305: error: Argument 1 to "_type_label" of "DatasetContract" has incompatible type "type[Any] | tuple[type[Any], ...] | None"; expected "type[Any] | tuple[type[Any], ...]"
  

  Mypy could not narrow the type of the attribute contract.expected_type when evaluated inside the if not type_matches: condition.
• Fix Applied: Bound contract.expected_type to a local variable expected and performed an explicit if expected is not None: guard before calling _type_label(expected):

  type_matches = True
  expected = contract.expected_type
  if expected is not None:
      type_matches = isinstance(value, expected)
      if not type_matches:
          if not collector.add(
              ValidationIssue(
                  "field.type",
                  (
                      f"Field {contract.name!r} has type {type(value).__name__}; "
                      f"expected {self._type_label(expected)}"
                  ),
                  ValidationSeverity.ERROR,
                  field=contract.name,
                  row_index=row_index,
                  value_preview=_safe_preview(contract.name, value),
                  constraint="expected_type",
              )
          ):
              return

• Outcome: mypy src/pyingestkit passes with 0 errors across 91 source files.

---

3. Documentation & CLI Configuration Fix — README.md

• Root Cause: Running the CLI quality demo jobs as documented in README.md failed with:

  Error: Invalid value for '--config' / '-c': File 'examples/plugin_package/demo-excel.yml' does not exist.
  

  The documentation in lines 132–134 erroneously specified demo-excel.yml, demo-ndjson.yml, and demo-parquet.yml instead of the actual config file demo-quality.yml.
• Fix Applied: Corrected README.md lines 132–134:

  pyingest run demo.excel_quality --config examples/plugin_package/demo-quality.yml
  pyingest run demo.ndjson_quality --config examples/plugin_package/demo-quality.yml
  pyingest run demo.parquet_quality --config examples/plugin_package/demo-quality.yml

• Outcome: All CLI commands in README.md execute successfully out-of-the-box.

---

Verification Matrix

Target Gate	Tool / Command	Status	Details
Functional Unit & Integration Tests	make test	PASS	152 passed in 1.70s
Lint & Formatting Check	ruff check && ruff format --check	PASS	168 files compliant
Static Type Analysis	mypy src/pyingestkit	PASS	91 files checked, 0 errors
Security Static Analysis	bandit -q -r src ...	PASS	0 issues detected
Vulnerability Audit	pip-audit	PASS	0 vulnerabilities found
Package Build	make build	PASS	SDist & Wheel generated
Full Source Gate	make verify	PASS	End-to-end clean run
Release Wheel Smoke Test	make release-check	PASS	Fresh virtualenv run over all 6 reference jobs

---

Summary of Reference Jobs Status

All six reference slices are 100% operational:

1. demo.local_file: Ingests local files into immutable RAW storage.
2. demo.http_csv: HTTP source -> Retry policy -> RAW provenance -> CSV parser -> Dataset validation.
3. demo.http_json: HTTP source -> Retry policy -> RAW provenance -> JSON parser -> Dataset validation.
4. demo.ndjson_quality: NDJSON parser -> Dataset -> Contract V2 -> Profile -> Quality reports.
5. demo.excel_quality: OpenPyXL parser -> Dataset -> Contract V2 -> Profile -> Quality reports.
6. demo.parquet_quality: PyArrow parser -> Dataset -> Contract V2 -> Profile -> Quality reports.

PyIngestKit v0.2.0 - Acquisition Release

PyIngestKit v0.2.0 — Acquisition Release

Official v0.2.0 release introducing the HTTP Acquisition & Dataset normalization pipeline:

• HTTP Source & Retries: Declarative HTTP client with exponential backoff and idempotency boundary (ADRs 022, 023).
• Dataset & Parsers: Standardized neutral Python Dataset container with CSV/JSON parsers (ADRs 025, 026).
• Provenance & Runtime: Comprehensive RAW artifact hashing, manifest recording and non-blocking validation observation (ADRs 024, 027).
• Reference Jobs: Three reference plugins (demo.local_file, demo.http_csv, demo.http_json).
• Quality & Security: Clean wheel verification, strict typing, and hardened release gates.

Attached Artifacts

• pyingestkit-0.2.0-py3-none-any.whl
• pyingestkit-0.2.0.tar.gz
• pyingestkit_demo_jobs-0.2.0-py3-none-any.whl
• pyingestkit_demo_jobs-0.2.0.tar.gz
• SHA256SUMS-v0.2.0.txt

v0.1.6 - Foundation Freeze

V0.1.6 Foundation Freeze

Stabilisation et Durcissement

• Moteur de persistance unifié : Adoption de SQLAlchemy 2.x Core pour SQLite et PostgreSQL, éliminant le SQL dynamique brut.
• Typage strict : Résolution complète de 11 anomalies Mypy en mode strict.
• Conformité & Style : Correction Ruff linter (PEP 484 explicit re-export) et uniformisation du formatage (105 fichiers).
• Sécurité : 0 alerte SAST Bandit, mise à jour des dépendances virtualenv pour 0 vulnérabilité (pip-audit).
• Gate de release : Introduction de make verify comme critère bloquant de gel Foundation.
• Documentation : Ajout des ADRs 018 à 021 et de la revue technique complète.

Règle Foundation : Le socle V0.1.6 est désormais gelé. Aucune modification ne doit être apportée à la Foundation lors du cycle V0.2 (Acquisition), sauf correction de bug critique avéré.