Add datatype field to Field and Metric; reframe is_time as role marker

[jonmmease]

AI Disclosure: I developed this PR and its description iteratively with Claude Code and Codex, and have reviewed the resulting changes fully.

Motivation

Closes #84. Related to #17 and follows the earlier discussion in #25.

The issue originally framed this as “use datatype rather than is_time.” After reviewing existing semantic-layer implementations, the cleaner model is to represent logical data type and temporal role independently:

• datatype says what values a field or metric produces.
• dimension.is_time says whether a field plays a temporal-dimension role.

For example, d_year, d_quarter_name, and d_month_name may have integer data types while all serving temporal roles. Snowflake Semantic Views and LookML similarly separate scalar representation from temporal classification.

So, this PR adds datatype alongside is_time, then carries that metadata through the Python bindings and existing converter paths that have native type information.

What changes

Core specification

Adds an optional top-level datatype property to both Field and Metric, backed by the following logical data types:

• String
• Integer
• Decimal
• Float
• Boolean
• Date
• Time
• DateTime
• DateTimeTz
• Opaque

The shared names align with the ontology specification’s built-in value types where possible. Time, DateTimeTz, and Opaque are additional types. I think these would make sense as ontology value types as well, but this PR doesn't modify the ontology’s layer.

The JSON Schema defines the authoritative DataType enum. Until the YAML specification is generated from JSON, spec.yaml mirrors the same enum explicitly.

Temporal role semantics

dimension.is_time remains valid and is documented as independent of datatype.

For fields with a dimension block:

1. An explicit is_time value wins.
2. When is_time is unset, it defaults to true for Date, Time, DateTime, and DateTimeTz, and to false otherwise.

A field without dimension metadata remains a fact regardless of its datatype. This supports combinations like:

• datatype: Integer with is_time: true for a year grain.
• datatype: DateTimeTz with is_time: false for a timestamp that should not be exposed as a time dimension.
• is_time: true without datatype for legacy models that know the role but do not declare a scalar type.

Python bindings

• Adds and exports OSIDataType.
• Adds optional datatype fields to OSIField and OSIMetric.
• Adds OSIField.is_time_dimension() to implement the effective temporal-role rule.
• Tests keep the Python enum synchronized with the core JSON Schema.

Existing converters

I updated the existing converters to propagate data types where it looked unambiguous. Here tagging some folks who have worked on the various converters for visibility.

• @sfc-gh-tgao — Snowflake
• @jaceksan — GoodData
• @jbonofre — Polaris
• @deutschn — Salesforce

[KSDaemon]

One concern here is that using a generic number type isn’t really a valid interoperability strategy. Different data warehouses interpret and implement it differently, so it’s not interchangeable in practice.

For example, you can’t take a model built against data in Snowflake and expect it to behave identically in Databricks just because both use a number type. As we’ve already discussed in other issues/threads, precision and scale vary across systems, and that directly affects results.

Without explicitly specifying those parameters, this approach is likely to introduce subtle bugs. If strict type checking is enforced and there’s no automatic conversion, things will simply break. Even with implicit or explicit casting at the physical layer, you can still end up with precision loss, which is not acceptable for many use cases.

So I think we need a different approach here. Either we make the type definition explicit (e.g., always specifying precision and scale for numeric types), or we introduce some abstraction that preserves correctness across backends instead of relying on a loosely defined number.

[jonmmease]

Thanks for the feedback @KSDaemon. The frame of reference I'm coming from is that these types need to be precise enough to generate SQL on top of (e.g. a BI tool building on top of the semantic definitions). In my experience, the SQL required for integers vs float/decimal can differ for warehouses that use truncation semantics for integers. I have not come across cases where I've needed to generate different SQL depending on whether the underlying type is a float or decimal.

But it is worth taking a step back to ask what else folks would rely on these logical data types for. Do you have any particular use cases in mind?

[jonmmease]

Following up. Are there cases beyond the generation of SQL for semantic queries that you have in mind to support here? I'm happy to add decimal as data type, but I'd like to understand a motivating scenario for how the spec would be used in such a way that number and decimal would be treated differently.

[jochenchrist]

Using JSON Schema data types (https://json-schema.org/understanding-json-schema/reference/type) as a minimum would be a good idea.

[xavipereztr]

Hi @jonmmease !

Thanks for your work on this PR! Some comments on this PR based on comments and discussions we have opened in regards of including spatial semantics in the OSI spec:

• Spatial dimension type: extending dimension with a spatial descriptor for geometry/geography and spatial index data #114
• Support field datatype rather than is_time #84

On geometry and geography as first-class datatype values

We'd like to make the case for adding geometry and geography to the
datatype enum rather than leaving spatial columns to fall through to other.

Both are first-class native types in the leading cloud data warehouses:

• Snowflake: GEOGRAPHY and GEOMETRY
• BigQuery: GEOGRAPHY (WGS84 point/line/polygon)
• Databricks: GEOMETRY
• PostGIS: geometry and geography

The geometry / geography distinction is also meaningful and standardized
(OGC/ISO SQL): geography operates on a spherical earth model with real-world
units, while geometry operates on a flat Cartesian plane. Consumers need to
know which they're dealing with to generate correct spatial SQL.

Leaving these as other + custom_extensions means every OSI implementation
dealing with spatial data falls back to vendor-specific metadata, something that at CARTO we believe it should not happen.

On dimension.spatial as the spatial role marker

The type/role split you've introduced gives spatial data a natural place to
live. We're working on a proposal (related to #69)
that would add a spatial object to the dimension block as the spatial
equivalent of is_time — a role marker that carries the metadata consuming
tools need: geometry type, SRID, spatial index system and resolution, and
geographic level. It would look like this:

- name: geom
  datatype: geometry              # proposed new enum value
  dimension:
    spatial:                      # proposed role marker + metadata
      type: polygon
      srid: 4326
      geographic_level: "census_block_group"
      geographic_hierarchy:
        parent: "census_tract"
        children: []
  description: "Block group boundary polygon (WGS84)"

- name: h3
  datatype: string                # H3 index is stored as a string
  dimension:
    spatial:
      type: spatial_index
      index:
        system: h3
        resolution: 8
        rollup_resolutions: [7, 6, 5]
      geographic_level: "h3_cell_res8"
  description: "H3 cell index at resolution 8"

This maps exactly onto the pattern you're establishing: datatype for the
data type question, dimension.spatial for the role/semantics question.
We can open a separate discussion on the spatial descriptor, but wanted to
flag it here so the geometry/geography datatype values are on the radar
before the enum is locked in.

On number precision — +1 to @KSDaemon's concern. We raised the same
point on issue #84 and in a comment there. Worth resolving before this merges.

[jonmmease]

Hi @xavipereztr, your descriptions of geometry and geography make good sense to me as additional data types, though I'd lean toward incorporating these as type additions in your proposal in #69 rather than this PR.

I'm not opposed to adding a decimal type, but I'd be interested to articulate a case where a consumer of the semantic model would act differently for a float vs decimal column or expression type.

You mentioned in the other issue:

So a value like 1.5 in a column declared as unqualified NUMBER/NUMERIC survives on BigQuery but is silently truncated to 2 on Snowflake and Databricks. Precision ceilings also diverge (29 vs 38 total digits).

If this value is in a column with a decimal type in BigQuery or Snowflake, it's either already truncated or the precision accommodates it. If I'm writing SQL generation logic on top of this dimension, are there cases where this needs to differ when the underlying type is decimal vs a float type?

Another factor would be, for translation purposes, whether any mainstream semantic layers already distinguish these numeric types (cube and lookml do not as far as I've seen). If so, that would also be good argument to separate decimal as a dedicated type.

Grounding this in a use case would also help me think through whether decimal by itself is useful without the specific scale and precision parameters, or whether these parameters would be necessary.

[xavipereztr]

Hi @jonmmease - on the geometry and geography aspect, are you suggesting we create a PR based on #69 or shall we wait for further discussion with OSI members?

On the numeric simplification, I guess that you are right that between integer vs number is largely enough for most use-cases.

[jonmmease]

I only meant that I think it makes sense to attach the additional geometry/geography data types to the dimension. spatial_data proposal in #69, and for these to be evaluated together through the OSI standards process (of which I'm not familiar with the details).

[jochenchrist]

maybe the default timestamp should be with tz (this is the most common use for timestamps to avoid typical errors). then Add a timestamp_ntz for the special case when really no timezone is expected.

[jonmmease]

I don't have a strong opinion on it, but I tend to lean towards postgres naming. But also not opposed to omitting timestamp for clarity and only having timestamp_tz and timestamp_ntz

[jonmmease]

Hey all, I'm not up to speed on the decision making process here, or who the decision makers are, but I'd like to do what I can to move this forward. I'm happy to make adjustments to the type system, but not sure we have consensus on this yet.

Would it be possible for someone with decision making authority here to summarize the desired changes?

[jonmmease]

closing in favor of apache/ossie-temp#12

[khush-bhatia]

Hey @jonmmease Sorry for the confusion here, now the ASF setup is fully done and the original repo is transferred to apache. With that all the open PRs are correct preserved. I am reopening this PR and reviewing now. The PR on the ossie-temp can be closed.

[khush-bhatia]

Could you please define the datatype enum in the yaml spec as well ?

I want to make the json spec the source of truth and have the yaml spec be generated via scripts, but until then we will need to preserve the spec in two places unforunately.

[khush-bhatia]

LGTM, could you please resolve the merge conflicts and one comment on left on keeping the yaml and json version of the spec in sync.

[khush-bhatia]

The datatypes for geometry/geography can be added in a follow up PR.

[Copilot]

Pull request overview

This PR introduces a portable logical datatype attribute for fields and metrics across the core spec, Python models, examples, and multiple converters, while reframing dimension.is_time as an independent temporal role marker (with defaulting behavior when is_time is unset).

Changes:

• Adds DataType (datatype) to the core schema/spec + Python models and updates examples/docs accordingly.
• Updates converters (Snowflake, Salesforce, Polaris, GoodData, dbt/MSI) to preserve/emit datatype appropriately while keeping is_time as a role signal.
• Expands/updates test coverage across Python and converter test suites for datatype mapping and effective time-role behavior.

Reviewed changes

Copilot reviewed 42 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
python/tests/test_models.py	Adds tests for schema alignment, serialization, and effective time-role defaulting.
python/src/ossie/models.py	Introduces OSIDataType, adds datatype to OSIField/OSIMetric, and adds OSIField.is_time_dimension().
python/src/ossie/init.py	Exposes OSIDataType in the public Python API.
examples/tpcds_semantic_model.yaml	Annotates example fields/metrics with datatype and demonstrates coexistence patterns.
docs/index.md	Updates glossary entry to document Field.datatype and its allowed values.
core-spec/spec.yaml	Documents datatype and clarifies dimension.is_time semantics/defaulting.
core-spec/spec.md	Adds DataType section, updates Field/Metric docs and examples, and documents type-vs-role guidance.
core-spec/osi-schema.json	Adds $defs.DataType and wires datatype into Field/Metric schema definitions.
converters/snowflake/tests/test_osi_to_snowflake_yaml_converter.py	Adds datatype mapping tests and classification tests for datatype+role interactions.
converters/snowflake/tests/example_converted_tpcds_semantic_model.yaml	Updates expected converted output to include Snowflake data_type emissions for fields.
converters/snowflake/src/osi_to_snowflake_yaml_converter.py	Adds datatype mapping to Snowflake data_type and updates _classify_field to respect explicit is_time then datatype defaulting.
converters/snowflake/README.md	Documents Ossie datatype → Snowflake data_type mapping and metric datatype omission rationale.
converters/salesforce/src/test/resources/examples/osiToSalesforce.yaml	Updates Salesforce example input to include datatype on fields/metrics.
converters/salesforce/src/test/java/org/apache/ossie/SalesforceToOsiConverterTest.java	Extends import tests to assert datatype mapping from Salesforce → Ossie.
converters/salesforce/src/test/java/org/apache/ossie/OsiToSalesforceConverterTest.java	Extends export tests to validate portable-vs-exact datatype precedence behavior.
converters/salesforce/src/test/java/org/apache/ossie/converter/SalesforceDataTypeMapperTest.java	Adds unit tests for Salesforce↔Ossie datatype mapping and compatibility logic.
converters/salesforce/src/main/java/org/apache/ossie/converter/SalesforceDataTypeMapper.java	Introduces centralized Salesforce datatype mapping/compatibility utilities.
converters/salesforce/src/main/java/org/apache/ossie/converter/MetricMappingHandler.java	Imports Salesforce metric result dataType into Ossie datatype when present.
converters/salesforce/src/main/java/org/apache/ossie/converter/FieldMappingHandler.java	Imports field datatype, uses mapper for temporal detection, and applies portable datatype on export with precedence rules.
converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java	Adds datatype constant and removes hard-coded Salesforce temporal constants.
converters/salesforce/README.md	Updates converter docs to describe direction-specific limits and datatype/role behavior.
converters/README.md	Documents datatype vs is_time split and adds converter guidance for both fields and metrics.
converters/polaris/src/test/java/org/apache/ossie/converter/polaris/OsiPolarisConverterTest.java	Updates Polaris tests for datatype support and exact Iceberg type round-tripping behavior.
converters/polaris/src/test/java/org/apache/ossie/converter/polaris/IcebergTypeMapperTest.java	Adds comprehensive tests for Iceberg↔Ossie datatype mapping and nested-ID reassignment.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisImporter.java	Maps Iceberg physical types to Ossie datatype, preserves exact Iceberg type in custom_extensions, and keeps temporal role classification.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisExporter.java	Resolves Iceberg types using (1) exact extension, (2) portable datatype, then legacy fallbacks; regenerates nested IDs for exported schemas.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OsiYamlGenerator.java	Emits datatype and field-level custom_extensions in generated Ossie YAML.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OsiModelParser.java	Parses datatype and custom_extensions for fields and datasets.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/model/OsiModel.java	Extends Polaris model to include datatype and customExtensions on fields.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/IcebergTypeMapper.java	Introduces shared mapping logic for Iceberg physical types, portable datatypes, exact-type extraction, and nested-ID reassignment.
converters/polaris/README.md	Updates mapping reference and documents datatype + exact-type preservation and precedence rules.
converters/gooddata/tests/test_roundtrip.py	Adds round-trip coverage for native/unknown/missing GoodData source types via datatype and extensions.
converters/gooddata/tests/test_osi_to_gooddata.py	Adds detailed tests for datatype mapping, warnings, and extension precedence on export.
converters/gooddata/tests/test_models.py	Adjusts model tests to reflect optional GoodData sourceColumnDataType.
converters/gooddata/tests/test_gooddata_to_osi.py	Adds tests for GoodData→Ossie datatype mapping and Opaque preservation behavior.
converters/gooddata/src/ossie_gooddata/osi_to_gooddata.py	Uses shared datatype mapping helper; keeps legacy date-dataset heuristic explicitly tied to explicit is_time.
converters/gooddata/src/ossie_gooddata/models.py	Makes source_column_data_type optional and updates (de)serialization accordingly.
converters/gooddata/src/ossie_gooddata/gooddata_to_osi.py	Adds datatype extraction, sets datatype when present, and preserves unknown types via Opaque + extension.
converters/gooddata/src/ossie_gooddata/datatype_mapping.py	Adds centralized GoodData↔Ossie datatype mapping + warning/precedence logic.
converters/gooddata/README.md	Documents datatype mapping and limitations (Float/Time/Opaque behaviors).
converters/dbt/tests/test_osi_to_msi.py	Adds tests for effective time-role defaulting via datatype + explicit opt-out.
converters/dbt/src/ossie_dbt/osi_to_msi.py	Switches time-dimension classification to use field.is_time_dimension().
.gitignore	Adds *.py[cod] ignore pattern.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jonmmease]

Thanks @khush-bhatia! I brought the changes I made in the ossie-temp back here. Main changes are:

1. I adopted the type system from the ontology layer (with a few extras that we could consider adding to the ontology value types as well)
2. I updated several of the converters to propagate types where it was relatively unambiguous how to do so
3. Added types to core-spec/spec.yaml as well

[khush-bhatia]

@jbonofre and @QMalcolm could one of you also review this PR since this involves changes to the spec ?

[jbonofre]

@khush-bhatia of course! I will! Thanks!