Skip to content

0.5.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@mattcasters mattcasters released this 30 Jul 20:36
· 140 commits to main since this release
v0.5.0
f9a4ebe

Updates in this 0.5.0 release since 0.4.0:

OpenLineage / Marquez lineage export (#101)

  • New workflow action Export data lineage emits model-derived OpenLineage COMPLETE events for DV/BV/DM tables (table + optional column lineage)
  • Destinations: folder (one JSON file per table + summary) and/or HTTP POST to OpenLineage endpoints (Marquez, Collibra OL-compatible)
  • Configurable job and dataset namespaces
  • Physical location facets: standard dataSource plus hop_location (database connection/schema/table, CSV/Parquet folder+mask, Iceberg catalog location)
  • Dimension aliases (role-playing dims) keep logical dataset identity and symlink to the shared physical dimension (e.g. d_shipping_date → d_date)
  • Marquez-safe dataSource.uri encoding (spaces / unresolved ${variables}) so failed POSTs no longer abort mid-export when Fail on HTTP error is enabled
  • Optional operational enrichment from load_pipeline_metric
  • Local stack: ./scripts/run-marquez.sh and scripts/docker/compose.marquez.yml (API :5001, UI :3001)
  • Retail sample: send-lineage-to-marquez.hwf
  • Documentation: docs/openlineage-export.adoc

Optional primary and foreign keys in model DDL (#92)

  • Two optional, default-off model configuration checkboxes on DV (.hdv), BV (.hbv), and DM (.hdm): Generate primary keys in DDL and Generate foreign keys in DDL
  • CREATE TABLE only (no ALTER retrofit of constraints on existing tables)
  • DV/BV/DM PK and FK rules per model type; FKs skipped on SingleStore and non-FK-capable engines
  • Multi-source hubs/links: serial workflow generation so sources for the same table no longer race under parallel bulk load
  • Link CDC target ordering for STRING/HEX hash keys matches Hop SortRows (collation-safe)

Resource definition validation and catalog-safe remediation (#83)

  • Design-time Validate sources options dialog (baseline, check axes, optional report path)
  • Validation results master-detail; remediation via Remediation proposals…
  • Length remediation expands DV satellite attributes, BV SCD2 mapped columns, and DM SQL-sourced columns from the catalog field length (catalog never rewritten)
  • Multi-table remediation package: SQL script + Hop workflow under the schema-remediation folder
  • Read-only Versions tree in the Data Catalog perspective
  • Documentation: docs/resource-definition-validation.adoc

Portability / UX

  • Issue #98: portable model and execution-map paths for cross-host projects
  • Dark-mode note fills (dark orange / dark gray)
  • Overview presentation and docs refresh

Full notes: CHANGELOG.md

Requires Apache Hop 2.18.1 and Java 21.

Install

  1. Remove any older plugins/misc/datavault/ install of this plugin.
  2. Unzip hop-datavault-0.5.0.zip into your Hop installation root (it expands under plugins/misc/datavault/), or copy hop-datavault-0.5.0.jar to $HOP_HOME/plugins/misc/datavault/.
  3. Restart Hop GUI / CLI.
  4. Confirm Data Vault Update, Business Vault Update, Validate resource definitions, and Export data lineage appear in the workflow action palette; .hdv / .hbv / .hdm open in the visual modelers.

On Hop 2.19+, you can also install from the Data Hopper community marketplace using hop-marketplace-repo.yaml (see repository README).

Assets 4
Loading