Skip to content

Add datatype field to Field and Metric; reframe is_time as role marker#11

Closed
vndrewlee wants to merge 2 commits into
apache:mainfrom
vndrewlee:add-datatype-to-spec
Closed

Add datatype field to Field and Metric; reframe is_time as role marker#11
vndrewlee wants to merge 2 commits into
apache:mainfrom
vndrewlee:add-datatype-to-spec

Conversation

@vndrewlee

@vndrewlee vndrewlee commented Jul 9, 2026

Copy link
Copy Markdown

AI Disclosure: I drove this PR and description below iteratively with Claude Code, and have reviewed it fully

Recreates open-semantic-interchange/ossie#113 on the Apache Ossie repository.

Motivation

Addresses open-semantic-interchange/ossie#84.

This PR is the outcome of the discussion I started in open-semantic-interchange/ossie#25 (converted to open-semantic-interchange/ossie#84), also related to open-semantic-interchange/ossie#17.

The issue originally framed this as "use datatype rather than is_time". After a bit more thought and investigation into existing semantic layers implementations, I think it makes more sense to add data types as a new field/metric attribute, and keep is_time as a role attribute. This fits cleanly with the example in this repo where d_year, d_quarter_name, d_month_name have different data types, but temporal roles. Snowflake Semantic Views (YAML form) and LookML both split the same two axes.

So this PR adds datatype alongside is_time and documents them as independent properties.

What changes

  1. New datatype enum on Field and Metric (optional, top-level). Values: string, integer, number, boolean, date, time, timestamp, timestamp_tz, other. Logical types, not SQL-physical. Use other plus custom_extensions for types outside the enum.

  2. dimension.is_time stays valid. It is documented as a temporal-role marker, independent of datatype. A field may carry datatype: integer with is_time: true (e.g. d_year), or datatype: timestamp with no is_time (a plain timestamp column), or is_time: true alone (legacy-style).

  3. Default rule for is_time. When unset, is_time defaults to true if datatype is one of date, time, timestamp, timestamp_tz, and false otherwise. Explicit is_time always wins. This lets authors opt a temporal-typed column (e.g. an audit created_at) out of time-dimension classification with is_time: false.

  4. Snowflake converter updated. _classify_field now honors explicit is_time first, then falls back to the temporal-datatype default. Nine new tests cover the datatype paths, the mixed-metadata case (d_year with datatype: integer and is_time: true), and the opt-out case. The five original is_time tests are unchanged, proving back-compat.

  5. Example updated. examples/tpcds_semantic_model.yaml demonstrates three coexistence patterns: datatype only (majority of fields), datatype plus is_time: true coexisting (d_year), and is_time: true only (legacy style, retained on two fields as a compatibility demonstration). 25 fields and 5 metrics gained datatype.

  6. Docs. converters/index.md explains the type-vs-role split and the converter classification rule; docs/index.md glossary mentions datatype on the Field entry. A new Datatype and is_time: type vs. role section in core-spec/spec.md covers the combinations, default rule, consumer guidance, and migration recipe, with a Precedent note citing Snowflake Semantic Views and LookML.

  7. Also added *.py[cod] to .gitignore for Python bytecode files.

Impact on existing implementations

The change is additive and backward-compatible:

  • Models with only is_time. Continue to validate and convert unchanged. The Snowflake converter's time_dimension classification is preserved for every is_time: true field, including non-temporal grain columns like d_year.
  • New models with only datatype. Work as expected. A temporal datatype on a field with a dimension block classifies as a time_dimension via the default rule.
  • Models that set is_time: false explicitly on a temporal-typed column. Classification is now dimension, not time_dimension. This is intentional: the author has opted out. The only way to reach this case before this PR was to author an unusual model where is_time: false was on a column that was implicitly temporal in the consumer's view. If such models exist, they will flip classification; we judged this the right behavior because ignoring an explicit is_time: false is what consumers had implicitly been doing, and that silent override is worse than the opt-out.

I didn't update the Snowflake exporter to add the data type itself, as Snowflake has the types of everything internally. If we were to add a Snowflake importer, it would be important to map the Snowflake data types onto these OSI data types.

Test plan

  • Run Snowflake converter tests: python -m pytest converters/snowflake/tests/test_osi_to_snowflake_yaml_converter.py
  • Validate examples/tpcds_semantic_model.yaml against updated schema
  • Review spec docs for Ossie naming consistency

jonmmease and others added 2 commits July 9, 2026 14:08
Introduces a top-level `datatype` on Field and Metric with a closed logical
enum: string, integer, number, boolean, date, time, timestamp, timestamp_tz,
other. Addresses issue #84.

`datatype` and `dimension.is_time` are independent and orthogonal:

- `datatype` declares the field's logical data type (casting/serialization).
- `is_time` is a temporal-role marker (time-series analysis, temporal
  filtering). A field with `is_time: true` may carry any `datatype` (e.g.
  integer for a year grain, string for a month name, date for a calendar
  date).

When `is_time` is unset, it defaults to `true` for temporal datatypes
(`date`, `time`, `timestamp`, `timestamp_tz`) and `false` otherwise.
Explicit `is_time` always wins, so authors can set `is_time: false` on
an audit `created_at` to keep it off the time axis.

Taxonomy and type/role split were chosen after benchmarking 14 peer
semantic layers and 5 portable type standards. Notable precedent:
Snowflake Semantic Views' YAML authoring form has a `time_dimensions:`
collection whose entries can carry any `data_type` (the published example
annotates `order_year` with `data_type: NUMBER`); LookML's `dimension_group`
accepts `date`, `datetime`, `timestamp`, `epoch`, and `yyyymmdd`.

Snowflake converter updated: `_classify_field` honors explicit `is_time`
first, then falls back to the temporal-datatype default. 9 new tests
cover the datatype paths and the mixed-metadata cases (`d_year` with
`datatype: integer` and `is_time: true`, audit timestamp opt-out, etc.).

tpcds_semantic_model.yaml demonstrates three coexistence patterns:
datatype-only, datatype + is_time, and is_time-only.

Also added .gitignore for __pycache__ directories.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Vendors enumeration block was introduced during cherry-pick conflict
resolution but was not part of apache/ossie#113.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vndrewlee vndrewlee closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants