[OSSIE][NVIDIA_GSF] Add bidirectional GSF semantic model converter - #247
[OSSIE][NVIDIA_GSF] Add bidirectional GSF semantic model converter#247liofnv wants to merge 5 commits into
Conversation
|
#275 is not a supersession of this PR. It's a copy of this branch's code re-committed under different authorship, based on an earlier state of this PR. This PR was updated in 791b440 to target GSF's actual model interchange contract ( This remains the active contribution and I'll keep it current. Happy to fold in the CI workflow idea from #275. |
There was a problem hiding this comment.
Thanks for this, it's a well-structured and impressively well-tested (nice touch running the official validate.py against generated output). A few robustness/maintainability notes inline. The two I'd prioritize before merge are the exact version pin and the preserved-SQL dialect-parse behavior; the rest are follow-ups or doc notes.
jbonofre
left a comment
jbonofre
left a comment
There was a problem hiding this comment.
It looks good to me. Just minor comments from my side.
Add offline Ossie-to-GSF conversion with a CLI, round-trip metadata preservation, checked-in fixtures, and official Ossie schema validation.
Make the extension namespace reflect the full product name while continuing to accept the legacy GSF identifier on input.
The converter previously targeted a provisional document shape. GSF now defines the interchange format in its model_interchange schema, so emit and consume that contract exactly: a data_layer catalog, a semantic_layer of terms, SQL attributes and custom analyses, and zones, all keyed by id. Ossie-origin entities get deterministic UUIDv5 ids so repeated conversions are stable, and a GSF-origin document keeps its live ids, catalog properties, SQL text and zones so a GSF round trip is lossless. Ossie edits stay authoritative: preserved SQL and relationship records are reused only while they still match the Ossie model. Date-part keywords are excluded from column extraction, since sqlglot parses the unit in DATEDIFF(day, ...) as an unqualified column and it would otherwise be imported as a physical catalog column. A GSF-sourced catalog is also treated as authoritative, so an unrecognized SQL identifier can never widen it. Verified against a live GSF graph: export, convert both directions and re-import reproduces 114 tables, 1717 columns, 17 terms and 33 SQL attributes with no ids lost or invented, and a second import is a no-op.
Accept any 0.2.x Ossie version rather than one exact string, so a patch or dev bump of the spec no longer rejects every model until a constant is hand-edited. Parse SQL across candidate dialects instead of the default parser alone, and treat SQL that no dialect can parse as opaque rather than aborting. Preserved GSF SQL carries whatever dialect its connection reported, so a model that imported cleanly could previously fail to export again. Treat a date-part keyword as a unit only when it occupies the unit argument itself, so a column genuinely named day or month is kept everywhere else in the same call. Drop LAST_DAY, TRUNC and EXTRACT from the function list, whose first argument is data. Label expressions with the source connection's dialect where Ossie names one, and read a preserved native snapshot through a single helper so a hand-edited extension fails alike in both paths.
791b440 to
d302a70
Compare
The spec gained an optional logical datatype on fields and metrics. Reduce a GSF column's physical type to that vocabulary on import, so NUMBER(38,0) becomes Integer and a type Ossie cannot name, such as VARIANT, becomes Opaque. On export, write a canonical physical type for a column the Ossie model introduces, and never override one preserved from a real GSF catalog, since GSF reports what the connection actually holds. The two mappings are inverses, so a declared datatype survives a full cycle. A computed field and a metric have no single column behind them and GSF stores no type for either, so theirs is still not carried.
d302a70 to
5a26fd7
Compare
|
Also rebased onto current main and picked up #113: datatype now maps to and from the physical type on the GSF column behind a field, with the two directions as inverses so a declared type survives a full cycle. A type outside Ossie's vocabulary becomes Opaque, and a type preserved from a live GSF catalog always wins over the logical one. Computed fields and metrics have no column and GSF stores no type for them, so theirs isn't carried, documented in the README. There's a test that reads the DataType enum from osi-schema.json so a future member can't be added without the mapping noticing. |
3 participants
Summary
Adds an offline, bidirectional converter between Apache Ossie YAML and NVIDIA GSF's semantic model YAML.
ossie-gsf export: Ossie → NVIDIA GSFossie-gsf import: NVIDIA GSF → OssieNVIDIA_GSFcustom-extension identifierWhy
GSF keeps its active semantic layer in Neo4j. It now supports exporting and importing that layer as a standalone YAML document, which gives Ossie a portable representation to convert against — versionable and reviewable, with no live GSF deployment involved.
Mapping
The converter emits and consumes GSF's
GsfModelDocumentcontract, whose roots aredata_layer,semantic_layer, andzones.columns_attributesentrysql_attributes.manualentrycustom_analysesentryjoins,foreign_keys, andsemantic_fksOssie-origin entities get deterministic UUIDv5 identifiers so repeated conversions are stable. A GSF-origin document keeps its live identifiers, catalog properties, SQL text, and zones, so a GSF → Ossie → GSF cycle is lossless. Ossie edits stay authoritative: preserved SQL and relationship records are reused only while they still match the Ossie model.
Testing
Notes
The physical catalog must already be ingested in the target GSF, and that database needs a configured connection, since GSF validates imported SQL against the connection's dialect.
Known limitations are documented in
converters/gsf/README.md, including one represented table per term and the loss of Ossie composite unique keys, which GSF records per column.The GSF-side import/export implementation has merged: https://github.com/NVIDIA/GSF/pull/134