Skip to content

Add THOUGHTSPOT to the Dialect enum and SKIP_SQL_VALIDATION - #351

Merged
jbonofre merged 1 commit into
apache:mainfrom
djwaldo:feat/thoughtspot-dialect
Sep 1, 2026
Merged

Add THOUGHTSPOT to the Dialect enum and SKIP_SQL_VALIDATION#351
jbonofre merged 1 commit into
apache:mainfrom
djwaldo:feat/thoughtspot-dialect

Conversation

@djwaldo

@djwaldo djwaldo commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Registers THOUGHTSPOT as an expression-language dialect, mirroring exactly what MDX, TABLEAU and MAQL already do.

ThoughtSpot's formula language is not SQL. It has its own syntax and its own column-reference form:

concat ( [ORDERS::Region] , ' - ' , [ORDERS::Segment] )
if ( [Revenue] > 0 ) then 'positive' else 'zero or below'

Two things follow. Dialect is a closed enum, so a THOUGHTSPOT entry fails schema validation outright. And validate_sql falls through to a default-dialect sqlglot parse for anything not in SKIP_SQL_VALIDATION, so such an expression would raise a parse error rather than being left alone — which is precisely why MDX, TABLEAU and MAQL are in that set.

Evidence

A semantic model carrying ThoughtSpot expressions on both a field and a metric, against validation/validate.py:

Before

Validation FAILED with 2 error(s):
  [Schema] ... fields -> 0 -> expression -> dialects -> 0 -> dialect:
      'THOUGHTSPOT' is not one of ['ANSI_SQL', 'SNOWFLAKE', 'MDX', 'TABLEAU', 'DATABRICKS', 'MAQL', 'BIGQUERY']
  [Schema] ... metrics -> 0 -> expression -> dialects -> 0 -> dialect:
      'THOUGHTSPOT' is not one of [...]

After

Validation PASSED

Existing python/ and validation/ suites: 19 passed.

Impact on existing implementations

None. No behaviour changes for any other dialect — the change is additive to a vocabulary. A consumer that does not know THOUGHTSPOT treats it as it would any dialect it does not implement.

Why five files

The dialect vocabulary is stated in five places, and they have to agree:

File Change
core-spec/ossie-schema.json the Dialect enum
core-spec/spec.md the dialect table
core-spec/spec.yaml the dialects list
validation/validate.py DIALECT_MAP entry (None) + SKIP_SQL_VALIDATION
python/src/ossie/models.py the OssieDialect enum

The last is easy to overlook — without it the JSON schema accepts the dialect while the shared Python models package still rejects it.

The DIALECT_MAP entry is strictly redundant, since SKIP_SQL_VALIDATION short-circuits before the lookup, but MDX/TABLEAU/MAQL are each in both and I have followed that pattern rather than diverge from it.

Possibly relevant beyond this dialect

In discussion #342 on shared filters, part of the argument turns on a parse-based join-path resolution strategy being unworkable for any dialect in SKIP_SQL_VALIDATION. That set currently has three members; this would make it four. Better known while that design is under discussion than after.

Related Issues

Context: #285 and #269 (ThoughtSpot converter). Announced on dev@ossie.apache.org: [DISCUSS] ThoughtSpot converter: scope, licensing, and where it should live, with a dedicated thread for this change.

Per CONTRIBUTING's specification-change process this is opened alongside the dev@ announcement, for the 7-day discussion window and a [VOTE].

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure
  • Spec changes have been discussed on the mailing list or in a linked issue
  • Breaking changes to the spec are clearly called out in the summary — there are none; the change is additive

Validation

  • Validation rules in validation/ are updated
  • New validation cases are covered by tests — no existing test references any specific dialect member; happy to add one if reviewers would like

Tests

  • All existing tests pass (19 passed)

Compliance

  • ASF license headers are present on all new source files — no new files
  • No third-party dependencies are added

ThoughtSpot's formula language is not SQL. It has its own syntax and its own
column reference form:

  concat ( [ORDERS::Region] , ' - ' , [ORDERS::Segment] )
  if ( [Revenue] > 0 ) then 'positive' else 'zero or below'

The Dialect enum is closed, so a THOUGHTSPOT dialect entry fails schema
validation outright today. And because validate_sql falls through to a
default-dialect sqlglot parse for anything not in SKIP_SQL_VALIDATION, such an
expression would produce a parse error rather than being left alone.

This is the same situation MDX, TABLEAU and MAQL are already in, and this change
mirrors exactly what they do -- a DIALECT_MAP entry of None alongside membership
of SKIP_SQL_VALIDATION. No behaviour changes for any other dialect.

Five files, because the dialect vocabulary is stated in five places:

  core-spec/ossie-schema.json   the Dialect enum
  core-spec/spec.md             the dialect table
  core-spec/spec.yaml           the dialects list
  validation/validate.py        DIALECT_MAP + SKIP_SQL_VALIDATION
  python/src/ossie/models.py    the OssieDialect enum

The last is easy to overlook: without it the JSON schema accepts the dialect
while the shared Python models package still rejects it.

Verified against a semantic model carrying ThoughtSpot expressions on both a
field and a metric. Before: two [Schema] errors, one per expression. After:
Validation PASSED. Existing python/ and validation/ suites: 19 passed.

Proposed on dev@ossie.apache.org and discussed in apache#285 / apache#269.
@jbonofre
jbonofre self-requested a review August 31, 2026 14:42
"Dialect": {
"type": "string",
"enum": ["ANSI_SQL", "SNOWFLAKE", "MDX", "TABLEAU", "DATABRICKS", "MAQL", "BIGQUERY"],
"enum": ["ANSI_SQL", "SNOWFLAKE", "MDX", "TABLEAU", "DATABRICKS", "MAQL", "BIGQUERY", "THOUGHTSPOT"],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok, but worth to note that some converters will "fail".

For instance in nvidia GSF, _pick_expression finds no ANSI_SQL and silently fails through to return expression[0]["expression"], writing the ThoughtSpot formula into a GSF model as a SQL expression (no warning).
I believe DBT and Honeydew converters will behave the same.

Before this PR, the schema gate rejected such a document. Now it is accepted and produces broken output two layers down. The root cause is that no converter has any machine-readable way to ask "is this dialect SQL?".

Nothing to consider soon.

@jbonofre
jbonofre merged commit 211ce66 into apache:main Sep 1, 2026
djwaldo added a commit to thoughtspot/thoughtspot-agent-skills that referenced this pull request Sep 2, 2026
…d) (#493)

apache/ossie#351 merged 2026-09-01, approved by the project's ASF mentor.
THOUGHTSPOT is in the Dialect enum, in SKIP_SQL_VALIDATION, and in OssieDialect.
Verified against origin/main.

The plan's Global Constraints said the opposite - never emit the dialect until
#351 merges - which was correct when written and is now inverted. Plans B, C and
D read that section, so leaving it would have had them build the ANSI_SQL
fallback path for a problem that no longer exists.

Also records that ANSI_SQL keeps a role, but a different one: per learnings P8 it
is emitted ALONGSIDE THOUGHTSPOT where the expression is portable, not instead of
it. The constant is renamed FALLBACK_DIALECT -> PORTABLE_DIALECT accordingly,
because the old name now means the opposite of what it does, and nothing consumes
it yet - Plan B would have been the first.

The task bodies still show the pre-merge snippets. Those are the historical brief
for an already-executed plan; the binding Global Constraints section is the part
Plans B-D read, and it now says so explicitly.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
WilliamKelley pushed a commit to hex-inc/apache-ossie that referenced this pull request Sep 3, 2026
ThoughtSpot's formula language is not SQL. It has its own syntax and its own
column reference form:

  concat ( [ORDERS::Region] , ' - ' , [ORDERS::Segment] )
  if ( [Revenue] > 0 ) then 'positive' else 'zero or below'

The Dialect enum is closed, so a THOUGHTSPOT dialect entry fails schema
validation outright today. And because validate_sql falls through to a
default-dialect sqlglot parse for anything not in SKIP_SQL_VALIDATION, such an
expression would produce a parse error rather than being left alone.

This is the same situation MDX, TABLEAU and MAQL are already in, and this change
mirrors exactly what they do -- a DIALECT_MAP entry of None alongside membership
of SKIP_SQL_VALIDATION. No behaviour changes for any other dialect.

Five files, because the dialect vocabulary is stated in five places:

  core-spec/ossie-schema.json   the Dialect enum
  core-spec/spec.md             the dialect table
  core-spec/spec.yaml           the dialects list
  validation/validate.py        DIALECT_MAP + SKIP_SQL_VALIDATION
  python/src/ossie/models.py    the OssieDialect enum

The last is easy to overlook: without it the JSON schema accepts the dialect
while the shared Python models package still rejects it.

Verified against a semantic model carrying ThoughtSpot expressions on both a
field and a metric. Before: two [Schema] errors, one per expression. After:
Validation PASSED. Existing python/ and validation/ suites: 19 passed.

Proposed on dev@ossie.apache.org and discussed in apache#285 / apache#269.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants