Skip to content

hop-datavault 0.6.0

Choose a tag to compare

View all tags
@mattcasters mattcasters released this 03 Aug 19:17
· 122 commits to main since this release
v0.6.0
4b2965e

hop-datavault 0.6.0

Requires Apache Hop 2.19.0 (or a recent 2.19.0-SNAPSHOT until GA) and Java 21.

Hop runtime: Hop 2.19.0 is not yet published to Maven Central. To use this plugin you must either build Hop from source (branch/tag aligned with 2.19) or download a recent CI snapshot of the Hop client:

Older Hop 2.18.x (including 2.18.1) is not supported for 0.6.0.

Downloads

Channel URL
GitHub Release (manual install) hop-datavault-0.6.0.zip
Data Hopper Nexus (Marketplace / Maven) hop-datavault-0.6.0.zip

Maven coordinates: org.apache.hop:hop-datavault:0.6.0 via the data-hopper-community marketplace repository.

Metadata note — linked tables vs reference tables

In 0.6.0, canvas cross-model pointers and hub aliases are stored as linked tables:

Concept tableType (saved) Java type Meaning
Linked table LINKED_TABLE DvLinkedTable Pointer / hub alias into another table or model (UI: Add Linked Hub / Link / Satellite)
Reference table REFERENCE DvReferenceTable Physical vault code/catalog table (ref_*), natural keys + load audit, no hub hash

Compatibility: existing .hdv files that used TABLE_REFERENCE still load (dual-read). On the next save, they are rewritten as LINKED_TABLE. This rename avoids confusion with physical Reference tables (#110). Docs: dv-cross-model-references.adoc.

Reference tables (#110)

  • First-class Reference table object on the raw Data Vault canvas: natural-key code/catalog tables with DV load audit columns (no hub hash keys or satellite hashdiff)
  • GUI: add/edit dialog, icon, help topic; load mode FULL_REPLACE (delete+insert) with database and CSV source pipeline builders
  • Catalog type, lineage, DDL, Data Vault Update execution, and multi-source workflow wiring
  • Integration suite tests/reference-table/ (golden ref_country two-wave load)
  • Design notes: docs/plans/dv-reference-tables-plan.md

Linked tables (rename from “table reference”)

  • Canvas pointers / hub aliases are linked tables (LINKED_TABLE / DvLinkedTable)
  • User language: Add Linked Hub / Linked Link / Linked Satellite; canvas badge (linked) (aliases stay (alias))
  • Dual-read of legacy TABLE_REFERENCE; save rewrites as LINKED_TABLE

Cross-engine ORDER BY COLLATE (#108)

  • Do not apply a SQL Server bridge collation (e.g. French_CI_AS) on PostgreSQL ORDER BY (or Postgres ICU/locale names on SQL Server) when source and target engines differ
  • Same-engine remediation (SQL Server↔SQL Server, PostgreSQL↔PostgreSQL) is unchanged
  • Fixes hub/satellite/link update SQL that failed with collation "French_CI_AS" for encoding "UTF8" does not exist on Postgres targets fed from SQL Server sources

Source modeler (.hsm) and composite feeds (#105)

  • New visual source modeler (.hsm): tables, PK/FK import, relationships, multi-table source queries, notes, ELK layout, undo/clipboard parity with other modelers
  • Query builder: joins (relationship or explicit keys), projection, WHERE, generation mode AUTO/SQL/PIPELINE, SQL preview and row preview
  • COMPOSITE catalog DV_SOURCE: publish queries with field layout + pointer to .hsm query; optional cached SQL
  • Hub / link / satellite pipeline builders consume composite sources (single-connection SQL subquery or Merge Join pipeline injection)
  • DV canvas action Compose multi-table source…; source-model toolbar/context Publish to catalog
  • Retail sample: retail-example/models/source-tables-crm.hsm (query All customer info → feed_customer_enriched)
  • Docs: source-modeler-overview.adoc

Project and metadata search (#106)

  • Opt .hsm / .hdv / .hbv / .hdm into Hop 2.19 project search via CAPABILITY_SEARCH and createSearchable()
  • Content analysers for models and plugin metadata (resource definition group, catalog connection, execution metrics, quality rule set)
  • Open search results navigate to the model and select/edit the matching table or source query when available
  • Docs: search.adoc

Architecture export to Draw.io (#104)

  • Derived SOLUTION, DATA inventory, and aggregated MODEL layer diagrams
  • MODEL export unions all .hdv → data-vault.drawio, all .hbv → business-vault.drawio, all .hdm → dimensional.drawio
  • CLI / action: hop architecture-export (--also-data, --also-models)
  • Docs: architecture-export.adoc

Hub aliases — same physical hub twice on a link (#103)

  • Same-model hub aliases (linked table with optional role hashKeyFieldName) so a link can participate one physical hub more than once
  • Canvas Add Hub alias; link dialog, DDL, load pipelines, special records, optional FKs
  • Integration: tests/hub-alias-role-playing/; retail sample hub_sales_rep + hub_secondary_rep + lnk_order_rep

Satellite parent key source fields (DV2 independent feeds)

  • Hub owns logical business keys and hash order; hub satellites may list ordered source field names (parentKeySourceFields) when names differ from hub BKs
  • Satellite dialog tab Parent key source fields
  • Docs: dv-satellite.adoc, dv-hub.adoc

Update resource definition group action

  • Workflow action Update resource definition group runs every DV / BV / DM model on a resource definition group
  • Optional Manage vault update metrics (Begin/End); open referenced models from the canvas
  • Docs: update-resource-definition-group-action.adoc

SQL Server string length / Unicode (issue #91 follow-up)

  • Model lengths remain characters; vault SQL Server DDL still creates UTF-8 VARCHAR(n×3)
  • Field-mapping overflow uses effectiveStringCapacity (capacity matches vault DDL)
  • Catalog/CRM staging CREATE does not apply vault UTF-8 length expansion

Hop 2.19 platform requirements

  • Development and runtime target is Apache Hop 2.19.0 (use 2.19.0-SNAPSHOT until GA)
  • Enables database-backed execution information (e.g. OPS hop_executions), BINARY hash key sorting (apache/hop#7346), and Hop Marketplace install
  • Markdown notes on DV/BV/DM model canvases (Hop 2.19 note API)
  • Apache snapshots repository updatePolicy=always for reliable CI resolution of Hop SNAPSHOTs

Stability and tooling

  • Canvas SVG icon cache / SWT handle-leak hardening (ModelGraphSvgIconCache; reflective SwtGc#getNativeGc for mixed hop-ui snapshots)
  • FILE catalog mkdir when PROJECT_HOME is unresolved
  • Docker test image freshness: rebuild docker-hop:latest when target/hop-datavault-*.zip is newer than the image

Full changelog: https://github.com/mattcasters/hop-data-vault/blob/v0.6.0/CHANGELOG.md

Assets 3
Loading