Skip to content

Lineage Foundations & True SQL Preview

Choose a tag to compare

View all tags
@ilona0925 ilona0925 released this 12 Nov 07:11
· 73 commits to main since this release
v0.3.0
b6a33c7

🚀 v0.3.0 – Lineage Foundations & True SQL Preview

Highlights

  • Full lineage model
    Introduced TargetDatasetInput and TargetColumnInput for dataset- and column-level lineage, enabling true derivation tracking across all layers.
  • Modularized target generation
    Cleaner, more maintainable architecture — dataset creation and lineage wiring are now clearly separated by layer.
  • Smart UNION logic
    Automatic multi-source handling in Stage datasets, with consistent column alignment and NULL padding.
  • Enhanced SQL Preview
    Real derivation SELECT statements built dynamically from lineage, fully formatted with a colorized UI output.
  • Improved naming resilience
    Surrogate and lineage keys remain stable even when dataset names change.
  • Documentation overhaul
    Added dedicated pages for the Lineage Model, Logical Plan, and SQL Rendering Pipeline.
    Refreshed README and Changelog for clarity and completeness.

Internal improvements

  • Refactored apply_all() into modular steps for clarity and maintainability
  • Improved error handling and automatic ordinal management
  • Removed obsolete fields (derived_from, etc.)
  • Updated mkdocs.yml and CI deployment workflow for automated documentation builds

🧪 Testing & Quality

Structured testing foundation

  • Introduced the first complete automated test framework for the metadata generation platform
  • Added dedicated runtests.py launcher for consistent execution across environments
  • Integrated realistic DB-based lineage tests (Raw → Stage → Rawcore)
  • Added logic-only tests for hashing, naming, and validators
  • Prepared SQL Preview test templates for the future rendering pipeline
  • Added new documentation: 🧪 Testing & Quality

Impact
This milestone establishes a robust foundation for test coverage — ensuring safe refactoring, reproducibility, and confidence in every release.

Next milestone

Version v0.4.0 will focus on:

  • Introducing the BizCore layer (user-defined transformations)
  • Incremental model support and materialization strategies
  • Interactive lineage visualization in the UI

Full Changelog: v0.2.6...v0.3.0

Assets 2
Loading