[Proposal] Dedicated thread for Downstream Agent Field Reports

[marcociav-exmergo]

Hi everyone,

it's Marco from Exmergo.

What are the community's thoughts on creating a dedicated discussion thread (or label) specifically for downstream agent "field reports" on Ossie usage in production?

Coding (or autonomous) agents will be using the spec quite heavily, and we've noticed they induce the "issue-sprawl" phenomenon: fragmented, hyper-specific issues that clutter the tracker.

To avoid this, I propose we start compiling Field Reports:
Write-ups of an agent's end-to-end attempt to build with Ossie, noting:

• Where it succeeds
• Where it gets confused
• Any workarounds required

The value prop:

1. A tracker only tells you what broke. A report tells you what building with the spec is actually like, including what worked and shouldn't change.
2. Issues extracted from them come with context: what was tried, what was ruled out, why it was filed.

I'll post our first field report as a comment below. Drafted by Claude Code (Opus 5) when building an Ossie semantic layer (mirroring one of our production dbt semantic layers). Verified by me against main.

Disclosure: the report was produced with dex, an open-source analytics engineering toolkit for agents that we maintain.

Would a centralized thread for this kind of feedback be useful for the maintainers and the broader community?

[marcociav-exmergo]

Ossie field report: mirroring a production MetricFlow layer

• Date: 2026-09-09
• Spec version: 0.2.0.dev0
• Schema we implemented against: core-spec/ossie-schema.json @
  b5da5d66f0da4a0cd3388d52201dbf5523221a77
• Findings re-checked against: main as of 2026-09-09, so this report
  describes the specification as it stands today rather than only the revision we
  pinned. Where the two differ, we say so.
• Agent: Claude Code (Opus 5)
• Consumer: dex 1.12.1, an open-source
  warehouse exploration and semantic-layer tool. It validates Ossie documents
  offline against a pinned copy of the schema and reads them as a semantic
  catalog.
• Warehouse: BigQuery
• What we built: one Ossie document holding 1 semantic model, 11 datasets,
  77 fields, 15 relationships and 27 metrics, mirroring an existing production
  dbt MetricFlow layer of 11 semantic models, 26 entity declarations, 44 declared
  dimensions, 22 measures and the same 27 metrics (22 simple, 5 ratio).
• Outcome: the mirror was completed. Nothing in the specification stopped it.

We publish a product-analytics semantic layer in MetricFlow and wanted the same
definitions available in a format that does not require our toolchain to read.
This report is what we learned translating a real layer, not a toy one, into
Ossie. Examples below use a neutral orders / customers domain throughout; the
structure is faithful, the names are illustrative.

We are a consumer, not a contributor to the spec, and we assume the choices below
were made deliberately and for reasons we cannot see. Where we say something is
missing, we mean we could not find a way to express it, and we describe what we
did instead. Where the specification already states its intent, we quote it and
argue about the consequence rather than the decision.

Summary

#	Finding	Spec surface	Impact
1	A metric may span datasets by design, and nothing states how it is evaluated	$defs/Metric, spec.md	high
2	custom_extensions.data is a string, so every structured payload is encoded JSON	$defs/CustomExtension	medium
3	label exists only on Field	$defs/Dataset, Metric, SemanticModel, Relationship	medium
4	The Dialect enum is closed, and nothing marks which values are SQL	$defs/Dialect	medium
5	source is a relation reference or a query, with no discriminator	$defs/Dataset	medium, not hit by us
6	A relationship's two column arrays have no equal-arity constraint	$defs/Relationship	low
7	No stated rule for composing multiple documents	document scope	low
8	Draft consumers have no in-band way to pin a schema revision	root version	low, a question rather than a complaint

Was anything blocking?

No. We want to be plain about that, because a report that inflates friction
into obstruction spends credibility the next report will need.

Every gap below was routed around, the document validated, and the layer is in
use. The cost was not that we could not finish. It was that structured
semantics became unstructured metadata: things our source format states in
typed fields now live in a vendor extension and in English prose inside
ai_context. A consumer that does not know our vendor extension gets a document
that is correct about identity and silent about how to use it.

If there is one signal to take from this report, it is that. Findings 1, 2 and 3
all end the same way: something that wants to be a typed field becomes prose in
ai_context, or JSON inside a string, or a sentence bolted onto a description.
That repetition is the shape of what is missing.

Findings

1. A metric may span datasets by design, and nothing states how it is evaluated

What we hit. This is the finding we would most like a maintainer to read.

The specification is explicit that this is intended: metrics "are defined at the
semantic model level and can span multiple datasets", and spec.md's own
cross-dataset example, avg_orders, references orders.amount and
customers.id. It is equally explicit that evaluation is out of scope: the spec
"does not define how metrics are evaluated at runtime".

Those two statements are individually reasonable and together they leave a hole.
Two of our 27 metrics are ratios of the form "average number of child records per
parent", which is the same shape as avg_orders. Evaluated naively across the
declared relationship, the denominator fans out by the number of children per
parent and the answer is wrong. Not an error, not a refusal: a plausible number,
silently.

What we did instead. An explicit warning, in prose, because there is nowhere
structured to put it:

- name: orders_per_customer
  ai_context:
    instructions: >-
      The numerator is on orders and the denominator on customers. Ossie defines
      no join planning, so evaluating this expression as written over a join of
      the two datasets fans the denominator out by the number of orders per
      customer. Aggregate each side to its own grain first, then divide.
  expression:
    dialects:
      - dialect: ANSI_SQL
        expression: COUNT(orders.order_id) / NULLIF(COUNT(customers.customer_id), 0)

What it costs. This is the most dangerous artifact we produced. The other
findings lose information; this one lets a valid document state something that
reads as executable and is not.

There is no way to work around it inside the format, because a metric has no
structured reference to anything. $defs/Metric carries name, expression,
description, datatype, ai_context and custom_extensions, and nothing else,
on main today. It cannot name the dataset it belongs to, and it cannot name a
field. So a consumer cannot be told which dataset a metric is scoped to, cannot be
told which field carries its time axis, and cannot be warned that a particular
metric is the cross-dataset kind. Detecting that case means parsing the expression
and comparing the identifiers against every dataset's fields, in a dialect the
format does not guarantee is SQL (finding 4). We do exactly that, and we would
rather not have to.

A direction. We are explicitly not asking for a query language. We
understand interchange metadata is the scope and that the runtime question is
being worked separately.

Something much smaller would help, and it is one field: let a metric name the
dataset (or the grain) each operand is evaluated at. That alone would let a
consumer distinguish a single-dataset metric it can safely render from a
cross-dataset one it should refuse, which is the outcome that matters. It would
also give metrics the structured anchor they currently lack, so a consumer could
find the candidate time fields itself by looking at the named dataset, rather than
being told nothing at all.

As it stands, the safest conforming consumer and the most useful one behave
differently on the spec's own example, and neither is wrong.

2. custom_extensions.data is a string, so every structured payload is encoded JSON

What we hit. $defs/CustomExtension requires vendor_name and data, and
types data as string, described as a "JSON string containing vendor-specific
data". We understand the intent: it keeps arbitrary vendor payloads out of the
core schema's constraints.

It is where the structural metadata of our source layer ended up, because the core
schema has nowhere else for it: which aggregate each metric is built from, which
field a metric aggregates over, and which upstream model sits behind each dataset.
All of it JSON, serialised into a YAML string, on every dataset and every metric
in the document.

What it costs. The extension mechanism is the pressure valve for everything
the core spec does not model, so it carries more weight than its shape suggests.
As a string it gives up schema validation of the payload even when a vendor is
willing to publish a schema for its own extension, produces unreadable diffs (one
long line per extension), and creates real quoting hazards where a payload
contains a quote character. The interoperability the extension point exists to
provide is weakened by every consumer inventing its own encoding inside the
string.

A direction. Allowing data to be oneOf string and object would preserve
the "no core constraints" property while letting vendors publish and validate
their own payload shape. If the string is deliberate for stable round-tripping
across YAML and JSON, that reasoning is worth putting in the spec text, because it
is not evident from the schema and the natural reading is that it was easier.

3. label exists only on Field

What we hit. $defs/Field has label, described as "Label for
categorization". Dataset, Metric, SemanticModel and Relationship have
description but no label. We re-checked this on main today and it is still
the case. Our source layer carries a short human display name on every node, which
is what a BI tool puts in a picker, distinct from the sentence or two of
explanation.

What we did instead. Kept labels on fields, and folded them into the
description everywhere else.

What it costs. A consumer building a UI over the document has no display name
for a dataset or a metric. It shows the identifier, or the first N characters of
prose. The asymmetry is also just surprising: it is the kind of inconsistency that
makes an implementer stop and re-read the schema to check they have not missed
something.

A direction. Add label to those four, purely additive and backward
compatible. This looks like an oversight rather than a decision, but we would
rather ask than assume.

4. The Dialect enum is closed, and nothing marks which values are SQL

What we hit. Two related issues in one place.

Closed. $defs/Dialect is a fixed enum, so a document using a dialect token
added after a consumer's pinned schema is refused outright rather than degraded.
This is not hypothetical: the enum held 7 values at the revision we implemented
against and holds 9 on main today, with SIGMA and THOUGHTSPOT added. A
consumer pinned to the older revision refuses a document that upstream considers
valid, and cannot tell that a dialect addition is why.

Unmarked. The enum is described as covering "SQL and expression language
dialects", and indeed contains SQL dialects (ANSI_SQL, SNOWFLAKE, BIGQUERY,
DATABRICKS) alongside languages that are not SQL (MDX, TABLEAU, MAQL). We
checked main: no property anywhere in the schema distinguishes the two. A
consumer that wants to parse an expression, resolve a column, or validate syntax
must carry a hardcoded list of which enum members are SQL. We carry one. It was
already stale, since SIGMA and THOUGHTSPOT arrived without us knowing which
kind they are.

This matters more than it looks, because reading a bare identifier out of a MAQL
expression as though it were a column name produces confident nonsense.

A direction. Mark SQL-ness in the schema so consumers can branch on data
rather than on a hand-maintained list. That alone would fix the more dangerous
half. Separately, it is worth deciding whether dialect needs an extension path,
given that a closed enum is a compatibility boundary that moves.

5. source is a relation reference or a query, with no discriminator

What we hit. Dataset.source is a plain string, described in the schema as
"Reference to underlying physical table/view (database.schema.table) or query".
Both forms are explicitly allowed and nothing distinguishes them.

We did not hit this in practice, since every dataset in our layer names a
relation. We raise it because our consumer had to take a position on it, and the
only safe position costs something: it declines to resolve a query-valued source
to a physical relation at all, which means such datasets get no column-level
validation and no data-classification screening.

What it costs. A consumer must guess, refuse, or ask. Guessing wrong in one
direction addresses a table that does not exist; in the other, it treats a query
as an identifier. For any consumer doing safety checks against real columns, the
ambiguity converts directly into either false evidence or absent checks.

A direction. A discriminator field, or two properties (relation and
query), would remove the guesswork. If the union is deliberate flexibility, spec
text saying "consumers must not infer which form a source takes" would at least
make the burden explicit and consistent across implementations.

6. A relationship's two column arrays have no equal-arity constraint

What we hit. $defs/Relationship constrains from_columns and to_columns
individually, each minItems: 1, but not against each other. A relationship whose
sides differ in length describes a join with no complete tuple to match on, and
validates cleanly. Still true on main.

What we did instead. Our consumer added its own rule and refuses such a
relationship. That is our rule, not upstream's, and we flag it as ours.

A direction. JSON Schema cannot express cross-array length equality directly,
so this belongs with the integrity rules applied outside the schema, next to the
existing endpoint-exists check.

7. No stated rule for composing multiple documents

What we hit. Validation is defined for one document. We could find no guidance
on combining several, validating them together, or whether semantic model names
must be unique across files.

Composing documents is our consumer's choice, and we want to be clear the
specification does not ask for it. But a layer large enough to split across files
is a normal thing to want, and a consumer that namespaces catalog entries by
semantic model name collapses two same-named models onto each other.

What we did instead. Our consumer requires semantic model names to be unique
across the configured set and refuses otherwise. Again, our rule, not upstream's.

A direction. Either state a composition rule, or state explicitly that
composition is out of scope and the contract belongs to the consumer. Either is
fine. The silence is what leaves implementations to diverge quietly.

8. Draft consumers have no in-band way to pin a schema revision

We want to be careful here, because the specification already answers the
obvious version of this complaint. spec.md says plainly: "DRAFT version
in development, schema may change before 0.2.0 is released", and "0.2.0.dev0
(Unreleased): In-development next minor release. Schema is mutable; do not depend
on this version in production." That is clear, deliberate, and fair warning. We
are not asking for stability during a draft.

What we hit. The consequence is still worth naming. Because version is
const: "0.2.0.dev0" and the schema is mutable under it, two schema revisions
that accept different sets of documents both declare the same version. We saw this
directly: the Dialect enum grew from 7 values to 9 with no change in the
declared version, so a document using SIGMA is valid against one revision and
refused by another, and nothing in either the document or the schema tells them
apart.

What we did instead. Our consumer pins the schema by SHA-256 of the file and
treats the declared version as informational. Upgrading the pin is a reviewed
diff rather than a version bump.

Two questions rather than a request. First: after 0.2.0, will the constant
move on every change that alters the set of accepted documents? If yes, this
disappears at release and needs nothing now. Second: would the draft benefit from
an optional revision identifier alongside the version, so that consumers who do
want reproducibility during the draft period have an in-band signal instead of
each inventing content hashing? We are content to be told the answer is "pin the
commit, that is what a draft means".

What went well

This is not politeness. These are the parts we would not want traded away in a
future revision.

Multi-dialect expressions are the right design, and the cost is proportional.
Being able to carry a portable ANSI_SQL spelling beside a warehouse-native one
meant our document is readable by a consumer that is not on our warehouse, without
giving up the native expression where it matters.

What we did not expect is how rarely we needed the second spelling. We emit a
BIGQUERY dialect only where it actually differs from the portable one, which is
10 of 27 metrics and none of the 77 fields: a bare column reference is the same
string in both, and the divergence is concentrated in conditional aggregates and
safe division. So the mechanism costs nothing on the majority of a real document
and is there for the minority that needs it. That is the right shape for an
optional feature, and of everything in the format this is what we would defend
hardest.

Dimension.is_time is the best-specified thing in the schema. It separates
the temporal role from the column's data type, states its default explicitly
(true for Date, Time, DateTime, DateTimeTz when unset, false otherwise),
and provides an explicit opt-out for audit timestamps that should not be treated
as time dimensions. We implemented against that description alone and then tested
our consumer against all five cases it implies, including the type-independent
role flag and the opt-out. Every case behaved as the description says it should.
That is what an unambiguous spec paragraph buys, and there should be more of them.
Finding 1 is, in effect, a request for this paragraph's equivalent on Metric.

Explicit relationships with ordered column pairs beat implicit joins, and we
have evidence. Our source format joins by shared entity name: declare the same
entity on two models and they become joinable. Translating that into explicit
pairwise relationships forced us to write down 15 joins that had previously been
implied. Verifying those declarations against the warehouse then surfaced a
genuine defect in our own pipeline: one declared join is not supported by the
data, and had been silently dropping rows from queries for months. The implicit
format had hidden it. The explicit one made it a line we could test.

ai_context as a first-class field is forward-looking. Having a designated
place for guidance addressed to an automated reader, distinct from the
human-facing description, is unusual and we think correct. Note that finding 1
describes us abusing it to carry semantics that should be typed, and we would
rather that pressure was relieved by new fields than by the field being removed.

primary_key and unique_keys are separate, independent claims. Modelling a
grain and further uniqueness constraints as distinct statements, both
composite-aware and order-preserving, is more precise than the single-key
assumption many formats make.

additionalProperties: false throughout. A misspelled property is an error
rather than silently ignored metadata. Our own document happened to validate on
the first attempt, so this never caught us out during authoring; we tested it
deliberately afterwards, with a misspelled labl on a field and primary_keys on
a dataset, and both were refused with the exact path to the offending key. For an
interchange format where a silently ignored property means silently lost meaning,
this is the right default.

The schema is small enough to read in one sitting. Around 11 KB. We
implemented against it by reading it, not by trial and error, and that is worth a
great deal for an interchange format.

Identity survived completely. Every one of our 27 metrics, 11 datasets and
their fields kept its name, its source, its expression and its documentation. The
losses in this report are all structural metadata about those objects. Nothing
was untranslatable.

What this exercise did not test

Reported as untested rather than left silent, since silence reads as coverage.

• No composite keys and no composite relationships. Every grain in our layer
  is a single column, so multi-column joins went unexercised.
• No non-SQL dialects. We emitted BIGQUERY and ANSI_SQL only, so the
  preservation behaviour for MDX, TABLEAU and MAQL is untested by us,
  despite finding 4 being about them.
• No query-valued sources, per finding 5.
• No multi-document layer. One file, one semantic model, so finding 7 is
  reasoned rather than experienced.
• One warehouse. We wrote portable ANSI_SQL but only ever executed the
  BIGQUERY spelling, so our portability claim is untested in the only way that
  would count.
• No consumer other than ours has read the document, so we cannot say whether
  it is interoperable in practice, only that it is valid.