v0.6.3 – Support Partial Upsert For landing Writing Path

Highlights

• Upgrade the dldb dependency from dldb-v1.1.2 to dldb-v1.1.6.
• Add optional dldb heartbeat configuration through GatewayConfig.
• Refine trainability detection:
  • Keep Claude Code sessions in one append-only chain across currentDate rollovers.
  • Normalize harness-managed current-date reminders only during fingerprint matching.
  • Bump update_is_trainable stage version from v7 to v8.
  • Bump landing_enrichment_pipeline version from v4 to v5.
• Add ETL task discovery, enqueueing, worker execution, status tracking, warnings, and richer execution reports.
• Add support for partial landing upserts with insert_missing=False, avoiding unnecessary reads and rewrites of wide payload columns.
• Harden deletion for HASH-partitioned tables by pruning non-materialized buckets before issuing delete operations.
• Improve Arrow/Pandas columnar extraction and reduce unnecessary data conversion overhead.
• Add bucket health checks and scheduled table-maintenance utilities.
• Refine landing archive and cleanup workflows.
• Improve ETL timing, report, and operational logging.

Compatibility

• Python: >=3.10,<3.13
• dldb: v1.1.6
• Existing full-row landing and serving upsert behavior remains unchanged by default.
• Partial landing upsert requires dldb v1.1.5 or later.

Installation

Install from the release tag:

pip install --upgrade \
  "wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@v0.6.3"

For Safactory integration, pin the dependency to:

wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@v0.6.3