Structured custom extension / properties (avoid opaque JSON strings)

[MikeCarlo]

Spec Change: structured custom extension / annotation data (avoid opaque JSON strings)

Summary

Propose a core-spec change so converter residue and vendor metadata can be
stored as structured key/value data instead of opaque JSON strings in
custom_extensions[].data that every consumer must special-case-parse.

Spec discussion only — no converter migrations in this PR.

Problem

In 0.2.0.dev0, each custom extension is:

custom_extensions:
  - vendor_name: POWER_BI   # open string
    data: '{"…"}'          # JSON string (see `$defs/CustomExtension`)

String-encoded data is awkward for authors and tools: values are not
first-class in YAML/JSON Schema, nested structure is hidden behind
JSON.parse, and round-trips depend on each consumer knowing a vendor’s
private blob shape.

Converters often land unmapped or partially mapped properties in
these blobs when there is no clear core field yet (or when mapping is
incomplete). Examples include metrics, synonyms / ai_context, and other
vendor attributes — the exact set varies by converter. The common pain is
the same: portable or semi-portable information is trapped in a string.

Refs: core-spec/ossie-schema.json ($defs/CustomExtension),
core-spec/spec.md (Custom Extensions).

Goals

• Schema-valid structured key/value (and nested) metadata without parsing
  a JSON string.
• Keep a clear line between vendor-private settings and any shared /
  portable residue.
• Prefer existing core fields when a property already has a home; use
  extensions / annotations only for genuine leftovers.
• Clear migration story under mutable 0.2.0.dev0.

Non-goals

• Removing custom_extensions.
• Enumerating or standardizing every vendor property in this PR.
• Shipping converter rewrites in this change.

Proposal

Proposal A - Object-valued data

Change data from string to object

custom_extensions:
  - vendor_name: SNOWFLAKE
    data:
      warehouse: ANALYTICS_WH
      some_unmapped_flag: true

This will break converters, however if a converter wants to continue to push just plain text into custom_extenstions -> data

custom_extensions:
  - vendor_name: SNOWFLAKE
    data:
      text: '{"…"}'          # JSON string Vendor choice to add or not
      warehouse: ANALYTICS_WH
      some_unmapped_flag: true

Proposal B - New properties map

Optional object on model / dataset / field / relationship / metric for
non-vendor or cross-tool residue; leave custom_extensions for
vendor-private blobs.

custom_extensions:
  - vendor_name: POWER_BI   # open string
    data: '{"…"}'          # JSON string (see `$defs/CustomExtension`)
  - properties:           # Proposed Addition
    import_source: fabric_lakehouse
    notes: "…"
    lineage-id: GUID

Clearer separation; new field to name and teach.

Recommendation

Go with Proposal A, Change data from string to object

Follow-ups (after agreement)

ossie-schema.json, spec.md, converters/README.md, examples,
validation / python models, then per-converter PRs.

[jeffbyk0]

I agree that this would a welcomed change. I recommend Proposal A.

[MikeCarlo]

@eisber and @m-kovalsky

I think this change would help us with the Power BI TMDL converter which lets us parse more of the extra meta data within the Semantic model that we would want to capture inside the custom_extensions data field.

Can you weigh in on this?

[jbonofre]

I will reply too.

[flyrain]

Thanks @MikeCarlo for the proposal. Option A is a clean solution to me by changing the data value from a string to an object. We don't have to specify any keys under data at this moment.

[MikeCarlo]

love this, and I agree, this will make our parsers much easier to rip through the OSSIE yaml.

it also does not block you from shoving raw text into a Text key inside the data object.. so not a blocker just a tad more structured

[jbonofre]

@MikeCarlo it would be great to also bring this discussion on the dev@ mailing list. As we say at Apache: "if it doesn't happen on the mailing list, it never happens".

Even if the GitHub discussions are "forwarded" to the dev@ mailing list, better to have direct discussions there.

[MikeCarlo]

superb idea, how do I do that, is there an email that I need to push this too.

I am planning on a spec change submission this week hopefully.

[jbonofre]

You can subscribe using dev-subscribe@ossie.apache.org (https://ossie.apache.org/community/). Thanks!

[jochenchrist]

There is a similar discussion for ontology in #146