Skip to content

🚀 elevata v0.7.1 - Materialization: Metadata-Driven Schema Evolution

Choose a tag to compare

View all tags
@ilona0925 ilona0925 released this 29 Dec 04:55
· 53 commits to main since this release
v0.7.1
0a3b373

This release introduces deterministic, metadata-driven schema evolution for target warehouses.
Structural changes are now applied explicitly, safely, and reproducibly — without breaking existing pipelines.

Schema evolution in elevata is no longer implicit or SQL-driven.
All physical changes are derived from metadata, validated via introspection, and executed in a controlled, lineage-aware manner.

✨ Highlights

🧱 Materialization & Schema Evolution

Target schemas are now synchronized against metadata definitions:

  • automatic provisioning of missing target tables
  • additive schema evolution via ADD COLUMN
  • deterministic reconciliation on full refresh
  • non-destructive behavior for incremental loads

Introspection is performed via SQLAlchemy, while all DDL execution is handled by existing execution engines — ensuring dialect consistency and operational safety.

🔁 Safe Column & Dataset Renames

Renames are now first-class, explicit operations:

  • former_names tracking on datasets and columns
  • physical RENAME TABLE / RENAME COLUMN instead of duplicate creation
  • lineage-aware, target-system independent behavior
  • guardrails for ambiguous or conflicting rename situations

This prevents silent schema drift and guarantees stable downstream references.

🕰 Rawcore History (_hist) Consistency

History datasets are now kept structurally aligned with their base tables:

  • entity and surrogate keys follow schema changes correctly
  • _hist tables are provisioned deterministically
  • structural drift between base and history datasets is prevented

Historization remains fully metadata-driven and reproducible.

🧾 Deterministic Load SQL

All generated write statements now explicitly list target columns:

  • no positional inserts
  • resilient to column additions and reordering
  • consistent across all supported dialects

This significantly improves robustness for evolving schemas.

🧪 Testing & Stability Improvements

  • extended unit tests for materialization planning and rename scenarios
  • improved DuckDB handling for introspection vs execution connections
  • removal of duplicate provisioning paths and race conditions
  • clearer separation between planning, provisioning, and execution

This release lays the architectural foundation for future validation, governance, and automation features — while keeping existing pipelines stable and predictable.

Full Changelog: v0.7.0...v0.7.1

Assets 2
Loading