hop-datavault 0.9.0

hop-datavault 0.9.0

Requires Apache Hop 2.19.0 (or a recent 2.19.0-SNAPSHOT until GA) and Java 21.

Highlights

• Shared model configuration (#126) — named Source / DV / BV / DM configuration metadata and Tools → Configure EDW setup
• Generate Data Vault from a source model (#125) — classify .hsm tables into hubs, satellites, links, and reference tables
• Optional orphan handling (#77) — PASS / INFER / SENTINEL / QUARANTINE / FAIL on child loads
• Hop Lineage View (.hlv, #79) — Marquez / folder / local-models backends, Show lineage, OPS overlay, Hop Web
• First-time EDW docs — architecture, getting-started build order, resource definition group, plugin HTML (docs/index.html) and toolbar EDW logo
• Load fixes — CSV satellite hash keys no longer pad integer business keys; SQL Server live schema resolves ${DB_NAME}

Install

Unzip hop-datavault-0.9.0.zip into $HOP_HOME (layout: plugins/misc/datavault/).

Optional DBeaver / JDBC client: add hop-hsm-jdbc-0.9.0.jar (zero third-party deps) as a driver.

Marketplace (after Nexus publish): org.apache.hop:hop-datavault:0.9.0

Nexus: https://repository.data-hopper.com/repository/hop-community-plugins/org/apache/hop/hop-datavault/0.9.0/hop-datavault-0.9.0.zip

Full notes: CHANGELOG.md

hop-datavault 0.8.0

hop-datavault 0.8.0

Requires Apache Hop 2.19.0 (or a recent 2.19.0-SNAPSHOT until GA) and Java 21.

Highlights

• Free SQL over source models (#117 / #115) — Apache Calcite free SQL against .hsm logical names, Source model SQL transform, Source model service metadata, Hop Server servlet /hop/sourceModelData, and thin hop-hsm-jdbc client for DBeaver (jdbc:hop-hsm://…)
• Source modeler pipeline sources (#116) — pipeline cards on .hsm, Record Definition Input (data), catalog PIPELINE loads, origin navigation, relationship lifecycle, retail ASN XML path
• Source JSON feeds (#114) — flatten JSON columns into catalog JSON sources; retail order-shipment path
• Metadata harvesting (#112) — harvest action, OPS history, schema gate HARVEST_RUN, harvest GUI, apply harvest / generate .hsm
• Update RDG load-overview — execution-id propagation and clearer ModelFailed reporting

Install

Unzip hop-datavault-0.8.0.zip into $HOP_HOME (layout: plugins/misc/datavault/).

Optional DBeaver / JDBC client: add hop-hsm-jdbc-0.8.0.jar (zero third-party deps) as a driver.

Marketplace (after Nexus publish): org.apache.hop:hop-datavault:0.8.0

Nexus: https://repository.data-hopper.com/repository/hop-community-plugins/org/apache/hop/hop-datavault/0.8.0/hop-datavault-0.8.0.zip

Full notes: CHANGELOG.md

hop-datavault 0.7.0

hop-datavault 0.7.0

Requires Apache Hop 2.19.0 (or a recent 2.19.0-SNAPSHOT until GA) and Java 21.

Highlights

• Optional load cycle ID (#111) — stamp LOAD_CYCLE_ID on DV / BV / DM tables; allocate per model update from a durable control table
• Composite hub business keys — multi-source-field → one vault BK column (VaultSpeed-style), with hash-over-parts defaults
• Optional satellite record-source column — omit physical source indicator for mono-source sats
• AI file-schema pack under docs/ai-file-schemas/
• Catalog / model-check / GUI polish (MODELS architecture export, Open in catalog, resource-group editor performance, and more)

Install

Unzip hop-datavault-0.7.0.zip into $HOP_HOME (layout: plugins/misc/datavault/).

Marketplace (after Nexus publish): org.apache.hop:hop-datavault:0.7.0

Full notes: CHANGELOG.md

hop-datavault 0.6.0

hop-datavault 0.6.0

Requires Apache Hop 2.19.0 (or a recent 2.19.0-SNAPSHOT until GA) and Java 21.

Hop runtime: Hop 2.19.0 is not yet published to Maven Central. To use this plugin you must either build Hop from source (branch/tag aligned with 2.19) or download a recent CI snapshot of the Hop client:

• Apache Hop 2.19.0-SNAPSHOT (hop-client)

Older Hop 2.18.x (including 2.18.1) is not supported for 0.6.0.

Downloads

Channel	URL
GitHub Release (manual install)	hop-datavault-0.6.0.zip
Data Hopper Nexus (Marketplace / Maven)	hop-datavault-0.6.0.zip

Maven coordinates: org.apache.hop:hop-datavault:0.6.0 via the data-hopper-community marketplace repository.

Metadata note — linked tables vs reference tables

In 0.6.0, canvas cross-model pointers and hub aliases are stored as linked tables:

Concept	tableType (saved)	Java type	Meaning
Linked table	LINKED_TABLE	DvLinkedTable	Pointer / hub alias into another table or model (UI: Add Linked Hub / Link / Satellite)
Reference table	REFERENCE	DvReferenceTable	Physical vault code/catalog table (ref_*), natural keys + load audit, no hub hash

Compatibility: existing .hdv files that used TABLE_REFERENCE still load (dual-read). On the next save, they are rewritten as LINKED_TABLE. This rename avoids confusion with physical Reference tables (#110). Docs: dv-cross-model-references.adoc.

Reference tables (#110)

• First-class Reference table object on the raw Data Vault canvas: natural-key code/catalog tables with DV load audit columns (no hub hash keys or satellite hashdiff)
• GUI: add/edit dialog, icon, help topic; load mode FULL_REPLACE (delete+insert) with database and CSV source pipeline builders
• Catalog type, lineage, DDL, Data Vault Update execution, and multi-source workflow wiring
• Integration suite tests/reference-table/ (golden ref_country two-wave load)
• Design notes: docs/plans/dv-reference-tables-plan.md

Linked tables (rename from “table reference”)

• Canvas pointers / hub aliases are linked tables (LINKED_TABLE / DvLinkedTable)
• User language: Add Linked Hub / Linked Link / Linked Satellite; canvas badge (linked) (aliases stay (alias))
• Dual-read of legacy TABLE_REFERENCE; save rewrites as LINKED_TABLE

Cross-engine ORDER BY COLLATE (#108)

• Do not apply a SQL Server bridge collation (e.g. French_CI_AS) on PostgreSQL ORDER BY (or Postgres ICU/locale names on SQL Server) when source and target engines differ
• Same-engine remediation (SQL Server↔SQL Server, PostgreSQL↔PostgreSQL) is unchanged
• Fixes hub/satellite/link update SQL that failed with collation "French_CI_AS" for encoding "UTF8" does not exist on Postgres targets fed from SQL Server sources

Source modeler (.hsm) and composite feeds (#105)

• New visual source modeler (.hsm): tables, PK/FK import, relationships, multi-table source queries, notes, ELK layout, undo/clipboard parity with other modelers
• Query builder: joins (relationship or explicit keys), projection, WHERE, generation mode AUTO/SQL/PIPELINE, SQL preview and row preview
• COMPOSITE catalog DV_SOURCE: publish queries with field layout + pointer to .hsm query; optional cached SQL
• Hub / link / satellite pipeline builders consume composite sources (single-connection SQL subquery or Merge Join pipeline injection)
• DV canvas action Compose multi-table source…; source-model toolbar/context Publish to catalog
• Retail sample: retail-example/models/source-tables-crm.hsm (query All customer info → feed_customer_enriched)
• Docs: source-modeler-overview.adoc

Project and metadata search (#106)

• Opt .hsm / .hdv / .hbv / .hdm into Hop 2.19 project search via CAPABILITY_SEARCH and createSearchable()
• Content analysers for models and plugin metadata (resource definition group, catalog connection, execution metrics, quality rule set)
• Open search results navigate to the model and select/edit the matching table or source query when available
• Docs: search.adoc

Architecture export to Draw.io (#104)

• Derived SOLUTION, DATA inventory, and aggregated MODEL layer diagrams
• MODEL export unions all .hdv → data-vault.drawio, all .hbv → business-vault.drawio, all .hdm → dimensional.drawio
• CLI / action: hop architecture-export (--also-data, --also-models)
• Docs: architecture-export.adoc

Hub aliases — same physical hub twice on a link (#103)

• Same-model hub aliases (linked table with optional role hashKeyFieldName) so a link can participate one physical hub more than once
• Canvas Add Hub alias; link dialog, DDL, load pipelines, special records, optional FKs
• Integration: tests/hub-alias-role-playing/; retail sample hub_sales_rep + hub_secondary_rep + lnk_order_rep

Satellite parent key source fields (DV2 independent feeds)

• Hub owns logical business keys and hash order; hub satellites may list ordered source field names (parentKeySourceFields) when names differ from hub BKs
• Satellite dialog tab Parent key source fields
• Docs: dv-satellite.adoc, dv-hub.adoc

Update resource definition group action

• Workflow action Update resource definition group runs every DV / BV / DM model on a resource definition group
• Optional Manage vault update metrics (Begin/End); open referenced models from the canvas
• Docs: update-resource-definition-group-action.adoc

SQL Server string length / Unicode (issue #91 follow-up)

• Model lengths remain characters; vault SQL Server DDL still creates UTF-8 VARCHAR(n×3)
• Field-mapping overflow uses effectiveStringCapacity (capacity matches vault DDL)
• Catalog/CRM staging CREATE does not apply vault UTF-8 length expansion

Hop 2.19 platform requirements

• Development and runtime target is Apache Hop 2.19.0 (use 2.19.0-SNAPSHOT until GA)
• Enables database-backed execution information (e.g. OPS hop_executions), BINARY hash key sorting (apache/hop#7346), and Hop Marketplace install
• Markdown notes on DV/BV/DM model canvases (Hop 2.19 note API)
• Apache snapshots repository updatePolicy=always for reliable CI resolution of Hop SNAPSHOTs

Stability and tooling

• Canvas SVG icon cache / SWT handle-leak hardening (ModelGraphSvgIconCache; reflective SwtGc#getNativeGc for mixed hop-ui snapshots)
• FILE catalog mkdir when PROJECT_HOME is unresolved
• Docker test image freshness: rebuild docker-hop:latest when target/hop-datavault-*.zip is newer than the image

---

Full changelog: https://github.com/mattcasters/hop-data-vault/blob/v0.6.0/CHANGELOG.md

0.5.0

Updates in this 0.5.0 release since 0.4.0:

OpenLineage / Marquez lineage export (#101)

• New workflow action Export data lineage emits model-derived OpenLineage COMPLETE events for DV/BV/DM tables (table + optional column lineage)
• Destinations: folder (one JSON file per table + summary) and/or HTTP POST to OpenLineage endpoints (Marquez, Collibra OL-compatible)
• Configurable job and dataset namespaces
• Physical location facets: standard dataSource plus hop_location (database connection/schema/table, CSV/Parquet folder+mask, Iceberg catalog location)
• Dimension aliases (role-playing dims) keep logical dataset identity and symlink to the shared physical dimension (e.g. d_shipping_date → d_date)
• Marquez-safe dataSource.uri encoding (spaces / unresolved ${variables}) so failed POSTs no longer abort mid-export when Fail on HTTP error is enabled
• Optional operational enrichment from load_pipeline_metric
• Local stack: ./scripts/run-marquez.sh and scripts/docker/compose.marquez.yml (API :5001, UI :3001)
• Retail sample: send-lineage-to-marquez.hwf
• Documentation: docs/openlineage-export.adoc

Optional primary and foreign keys in model DDL (#92)

• Two optional, default-off model configuration checkboxes on DV (.hdv), BV (.hbv), and DM (.hdm): Generate primary keys in DDL and Generate foreign keys in DDL
• CREATE TABLE only (no ALTER retrofit of constraints on existing tables)
• DV/BV/DM PK and FK rules per model type; FKs skipped on SingleStore and non-FK-capable engines
• Multi-source hubs/links: serial workflow generation so sources for the same table no longer race under parallel bulk load
• Link CDC target ordering for STRING/HEX hash keys matches Hop SortRows (collation-safe)

Resource definition validation and catalog-safe remediation (#83)

• Design-time Validate sources options dialog (baseline, check axes, optional report path)
• Validation results master-detail; remediation via Remediation proposals…
• Length remediation expands DV satellite attributes, BV SCD2 mapped columns, and DM SQL-sourced columns from the catalog field length (catalog never rewritten)
• Multi-table remediation package: SQL script + Hop workflow under the schema-remediation folder
• Read-only Versions tree in the Data Catalog perspective
• Documentation: docs/resource-definition-validation.adoc

Portability / UX

• Issue #98: portable model and execution-map paths for cross-host projects
• Dark-mode note fills (dark orange / dark gray)
• Overview presentation and docs refresh

Full notes: CHANGELOG.md

Requires Apache Hop 2.18.1 and Java 21.

Install

1. Remove any older plugins/misc/datavault/ install of this plugin.
2. Unzip hop-datavault-0.5.0.zip into your Hop installation root (it expands under plugins/misc/datavault/), or copy hop-datavault-0.5.0.jar to $HOP_HOME/plugins/misc/datavault/.
3. Restart Hop GUI / CLI.
4. Confirm Data Vault Update, Business Vault Update, Validate resource definitions, and Export data lineage appear in the workflow action palette; .hdv / .hbv / .hdm open in the visual modelers.

On Hop 2.19+, you can also install from the Data Hopper community marketplace using hop-marketplace-repo.yaml (see repository README).

0.4.0

Updates in this 0.4.0 release since 0.3.0:

Source-to-target lineage (#97)

• Derive table- and field-level lineage for DV / BV / DM with structured reason codes
• Lineage tab on table dialogs; explainable DDL on update actions and Generate DDL
• Catalog lineage siblings under hop/{project}/lineage/{dv|bv|dm}/{model}/
• Lineage drift gate on Validate resource definitions (blocking renames without explicit naming)
• Reverse lineage browser (Browse lineage… on resource definition groups)
• Fix Explorer multi-pane open for model files; TableView sort-safe open in reverse browser
• Documentation: docs/source-to-target-lineage.adoc

Data Vault / Business Vault / dimensional

• Issue #90: transactional links via dependent child keys
• Issue #81: separate DV and BV databases for incremental SCD2 and PIT
• Issue #91: expand SQL Server UTF-8 VARCHAR lengths for multi-byte data
• Issue #87: pipeline wall-clock timing on load_pipeline_metric
• Issue #85: pure Type 2 BV SCD2 and hybrid Type 1/2 guidance

Operations / build

• Jenkins/catalog fixture build fixes
• Reduced init logging

Full notes: CHANGELOG.md

Requires Apache Hop 2.18.1 and Java 21.

Install hop-datavault-0.4.0.zip by unzipping into the plugins folder of your Hop distribution (remove older versions of this plugin first).

0.3.0

Updates in this 0.3.0 preview release since 0.2.0:

• Issue #76 : CI/CD schema validation gate and downstream impact simulator
  • Catalog version tags (FILE catalog catalog-versions/, Tag / List from resource definition group)
  • Schema impact simulation (LIVE_SOURCE, WORKING_VS_VERSION, VERSION_VS_VERSION)
  • Validate resource definitions workflow action (schema gate)
  • Failure severity (FAIL_ON_BLOCKING, FAIL_ON_WARNINGS, WARN_ONLY)
  • Markdown/HTML validation reports with blast radius
  • Downstream impact graph (Source → DV → BV → DM)
  • GUI: validate results, issue detail, remediation proposals
• Retail example runtime hygiene
  • Gitignored work/ tree (edw-catalog, reports, HEMs, metrics)
  • Schema-gate baseline fixture v1.0.0 + bootstrap script
  • Stable load-e2e-sources-to-crm.hpl via ${RETAIL_CSV_WAVE}
• Documentation and screenshots for catalog versions and the schema gate

Also included since 0.2.0 (already on main):

• Issue #73 : Unicode-capable EDW targets / SQL Server UTF-8 VARCHAR
• Issue #57 / #42 : views and lineage starters
• Dialog and packaging polish

Requires Apache Hop 2.18.1 and Java 21.

Install hop-datavault-0.3.0-SNAPSHOT.zip by unzipping into the plugins folder of your Hop distribution (remove older versions of this plugin first).

0.2.0

Updates in this release since 0.1.0:

• Issue #47 : Incremental loading for Business Vault SCD2
• Issue #62 : Primary key handling, import and detection; open execution map from action
• Issue #63 : SQL Server collation remediation, error checking, and duplicate hub/satellite entries
• Issue #66 : Data profiling / data quality validations (Phase 1 + Phase 2)
  • Quality measure and gate foundation
  • Profile rules: NULL_RATIO_MAX, distinct bounds, REGEX, value length
  • SQL_ASSERTION rule (SELECT-only)
  • Persist quality runs, findings, and profile snapshots to dv_ops
  • ALERT_ONLY sinks (log + ops table)
  • Catalog Quality tab history browser
  • Preserve qualityRules and validation acks on model publish
  • Retail example post-update quality on published vault targets
• Issue #68 : Multi-DB integration, ops schema defaults, PIT, collation

Also included:

• Metrics insight threshold moved into the Vault metrics profile
• Open Markdown/HTML report from End Vault Update action
• Tutorial video links, images, texts, and scripts
• i18n fixes for missing/misplaced keys
• APL2 header cleanup

Install hop-datavault-0.2.0-SNAPSHOT.zip by unzipping into the plugins folder of your Hop distribution (remove older versions of this plugin first).

0.1.0

Updates in this release since 0.0.16:

• Issue #46 : Collect metrics from model updates and offer advice

• Issue #48 : Link tables in data vault models

• Issue #49 : Renamed "Get keys" button

• Issue #50 : multi select in data catalog and import improvements

• Issue #51 : Better catalog presentation

• Issue #52 : Create an action to generate an execution map

• Issue #54 : Mandatrory fields in Hub / Sat / link

• Issue #55 : live updates of updating models with stall detection

• Issue #56 : Provide a help button in all dialogs (and validate buttons)

• Issue #59 : Links docs not clear Or implicit mapping is not working

• Issue #60 : HPL link is not generating key correctly

• Issue #61 : Create a coaching panel in the modelers

• Retail example: data update fix

• Viewer improvements

• ShowRowsDialog

• Execution Information Location integration for AI advisor

• AI markdown viewer improvements

• AI Assistant settings can now switch between providers

• Improved help pages for link tables and commonmark for HTML rendering of markdown

Install hop-datavault-0.1.0-SNAPSHOT.zip by unzip-ing in the plugins folder of your Hop distribution (remove older versions of this plugin first).

0.0.16

Here are the new things since 0.0.14:

• PIT tables in the business vault
• SQL formatting in database source SQL widgets
• Added support for dimensional modeling
• Dimensional modeler Date dimension generator
• AI helper for BV/DM
• Merge Rows Plus transform
• Degenerate/Range dimension support
• Hop Execution Map generator and viewer
• Get a report of all changes in the source systems
• Docs update

The associated retail-example-0.0.16.zip download contains an Apache Hop project with different models and integration tests after initial load and updates. Please check file README.md to get started.

Install hop-datavault-0.0.16-SNAPSHOT.zip by unzip-ing in the plugins folder of your Hop distribution (remove older versions of this plugin first).