[OSSIE][DATABRICKS] Add Java Converter and Reformat - #333
Open
Haoranli503 wants to merge 7 commits into
Open
Conversation
jbonofre
self-requested a review
August 27, 2026 11:51
jbonofre
force-pushed
the
databricks-converter-v2
branch
from
August 28, 2026 05:09
366cccf to
5572106
Compare
Member
|
@Haoranli503 I pushed some fixes here. Can you please take a look? |
Contributor
Author
|
Hi @jbonofre , fixes look good. Thanks! I also pushed one commit addressing some issues we found when we are building native sql support based on the converter. Hope you can review those as well. Also feel free to change. |
Contributor
Author
|
Thanks @Haoranli503! |
17 tasks
Move the Python converter under converters/databricks/python/ (content unchanged) and add a Maven Java module -- library, CLI (OssieDatabricksConverter), JUnit tests, and fixtures under java/, package org.apache.ossie.converter.databricks -- as the maintained implementation. Add a root README describing the two-language layout, and a Java build job (mvn -B verify, JDK 21) in converter-databricks-ci.yml, mirroring the polaris converter. Signed-off-by: Haoran Li <haoran.li@databricks.com>
Replace the Python README's forward-looking 'Future effort' section with a deprecation note: the Java converter under java/ is the maintained implementation; the Python copy is kept for reference and no longer actively extended. Signed-off-by: Haoran Li <haoran.li@databricks.com>
Build: - maven-shade no longer writes dependency-reduced-pom.xml into the module root, where apache-rat failed `mvn verify` on it as an unapproved file - configure surefire to include **/*Suite.java: the default includes match none of the test classes, so the build ran zero tests and still passed - drop the **/*.md rat exclude and restore the ASF header on both READMEs - align snakeyaml with the 2.3 that jackson-dataformat-yaml declares Converter: - qualifyMeasure matches the whole qualifier run and resolves it from the leaf, so an expression that already carries a join path is no longer qualified a second time (SUM(customer.customer.region.population)) - de-alias measure qualifiers on import, the inverse of the export rewrite and what resolveColumn already did for dimensions - match dropped names outside string literals when cascading drops, so a name that only occurs in a literal no longer drops an unrelated column - quoteReplacement the stash unicode-escape pass, which halved an escaped backslash run instead of re-emitting it verbatim - notice the ai_context object members and the foreign-vendor extensions dropped from a field or a metric - validate a join source on import with the rule the export applies, so a view that imports cleanly is always exportable again CLI: - name the directions from the Apache Ossie model's point of view, matching the library Javadoc and the Python CLI: export = Ossie -> Metric View - give each command its own selector flag instead of sharing one field, and resolve the command before parsing arguments so --help prints usage - print to stdout without the extra newline, so stdout and -o agree Tests: - generate join-qualified and nested-path measures, and one_to_many branches, in the property round-trip suites - add a regression test per fix, plus a CLI suite (the CLI had none)
Signed-off-by: Haoran Li <haoran.li@databricks.com>
…und trip A dimension's display_name maps to the Ossie Field label, but the Ossie Metric schema has no label, so a metric view measure's display_name was dropped in the MV -> Ossie -> MV round trip. Preserve it in the DATABRICKS custom_extensions stash (the same mechanism as format/window) in MetricViewToOssie.convertMeasure, and restore it in OssieToMetricView.convertMetric. Ports apache#326 (landed in Databricks runtime as databricks-eng/runtime#251390). Co-authored-by: Isaac <no-reply@databricks.com>
… complex_joins stash A Metric View join whose `on` is not an equi-join of simple `alias.column` pairs (a non-equi operator, a SQL-function-wrapped key, or an extra filter predicate) has no Apache Ossie relationship form: the relationship schema requires from_columns/to_columns. The converter used to abort the whole MV -> Ossie conversion on such a join. Instead of aborting, preserve the join under the model's DATABRICKS custom_extensions (complex_joins) and warn, rather than emitting a schema-invalid stub relationship with no columns. The reverse converter merges the stashed joins back into the relationship graph and restores each raw `on` verbatim, so such a metric view round-trips (nesting and one_to_many included). Condition-less (cross) joins still have no Apache Ossie representation and are still rejected. Also preserve an equi-join's original `on` verbatim when rebuilding it from the from/to columns would not reproduce it (a fact side qualified by the source table name rather than `source`, or an `on` over equal columns that would rebuild as `using`); canonical joins stash nothing. Ports apache#321 (landed in Databricks runtime as databricks-eng/runtime#251398). Co-authored-by: Isaac <no-reply@databricks.com>
Haoranli503
force-pushed
the
databricks-converter-v2
branch
from
September 4, 2026 17:38
9b270b8 to
7d94fc6
Compare
…g tables Upstream apache#356 clarified the mapping table (a measure's display_name has no `label` on the Apache Ossie metric shape, so it rides in the DATABRICKS stash). The converter restructure turned the top-level README into a short pointer, so carry that clarification into the mapping tables now in python/README.md and java/README.md. Both converters stash a measure's display_name (Python via apache#356, Java via the measure display_name round-trip commit), so the tables match the behavior. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
Hi @jbonofre , all fixes are in. Can you help tal? Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Java implementation of the Databricks converter alongside the existing Python
reference, and restructures
converters/databricks/to host both languages. The Java library isthe implementation maintained going forward; the Python converter becomes a reference copy slated
for deprecation.
Conversion is unchanged in spirit: pure YAML text in, YAML text out, reading and writing the two
formats as parsed maps and lists. The Java and Python implementations target the same Databricks
Unity Catalog Metric View YAML v1.1 contract, and the checked-in fixtures pin the expected output
for both directions (
export: Ossie -> Metric View,import: Metric View -> Ossie).Changes:
src/,tests/,pyproject.toml,README.md,uv.lock) moves underconverters/databricks/python/with no content changes,making room for the Java tree.
java/— the bidirectional converter as a Maven module: library, anOssieDatabricksConvertercommand-line tool, JUnit 5 tests, and YAML fixtures. Packageorg.apache.ossie.converter.databricks, following the conventions of the existingpolarisandsalesforceJava converters (parentorg.apache:apache:39; deps SnakeYAML / Jackson / JUnit 5;apache-rat-pluginlicense audit; shade plugin for the executable CLI jar).README.mddescribing the two-language layout and pointing at eachimplementation's own README.
.github/workflows/converter-databricks-ci.yml) — point the Python job at the newpython/working directory, and add a Java job (mvn -B verify, Temurin JDK 21) mirroringconverter-polaris-ci.yml.Related Issues
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructs or converter supportTests
pytest/ CI green)Compliance