Skip to content

feat(messagedefs): authorable message definition collections - #380

Merged
ako merged 7 commits into
mainfrom
feat/message-definitions-272
Sep 3, 2026
Merged

feat(messagedefs): authorable message definition collections#380
ako merged 7 commits into
mainfrom
feat/message-definitions-272

Conversation

@ako

@ako ako commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Closes #272. Implements PROPOSAL_authorable_message_definitions.md (#371, #375).

A mapping binds to one of four schema sources, and MDL could create one. Message
definitions are 74 of the 327 mappings in the demo corpus (22.6%) and were
the only other one a script could bring into existence — an XML schema holds an
imported .xsd and a web service holds a WSDL, but a message definition holds
nothing external: it is a selection over the domain model.

create message definition collection Sales.MD_Order
  folder 'Messages'
(
  definition OrderMessage for Sales.Order as 'Orders' (
    OrderId,
    TotalAmount as 'Total',
    Sales.OrderLine_Order/Sales.OrderLine as 'Lines' ( Sku, Quantity ),
    Sales.Order_Customer/Sales.Customer ( FirstName, Address example 'Kerstraat 5' )
  )
);

create import mapping Sales.IMM_Order
  with message definition Sales.MD_Order.OrderMessage
{ create Sales.Order { OrderId = OrderId } };

Plus DROP, DESCRIBE (re-executable), SHOW — there was no listing at all —
and targeted ALTER for both definitions and members.

The evidence this is right

describeexec against ako/TestApp's hand-authored collection reproduces
Studio Pro's document exactly.
The executor prints Unchanged, meaning
canon.Reconcile found the rebuilt content semantically equal and elided the
write. mxbuild 11.13/11.14: 0 errors, including mappings in both directions
bound to an authored definition.

One hand-authored document beat a 4,686-element census

The proposal was measured over 36 marketplace-module collections. Round-tripping
one hand-authored document (thank you) found five things that census had not:

  1. Path is a chain of ORIGINAL names, not exposed ones, and an
    association contributes two segmentsOrder|OrderLine_Order|OrderLine|Amount.
    Confirmed afterwards at 4,707/4,707 once we knew to look.
  2. Typed-array marker is 2, where the codec defaults to 3.
  3. Every element serializes Children even when empty — the bare [2],
    same MandatoryLists rule as a rule document's Flows.
  4. PrimitiveType is mapped, not passed through: Long → Integer,
    AutoNumber → Integer, Enumeration → String. A pass-through gets 279
    corpus elements wrong; TestApp caught it on a Long.
  5. Example is author-set — empty in 4,686/4,686 of the corpus, set in
    TestApp. Hardcoding it empty would silently drop the one that exists, so
    example '...' is now syntax.

The round-trip test is what finds these, and the baseline matters: compare
against the stored bytes, not a re-encoding of the decoded original — a
lazily-decoded element that was never marked dirty encodes as an empty document,
so that baseline passes by comparing nothing to nothing.

The derivation most likely to be got wrong

MaxOccurs on an association follows the direction of traversal, not the
association's type — all 927 resolvable associations in the corpus are
Reference, yet 526 store 1 and 401 store -1:

holder is cardinality count
the association's FROM entity (following the FK) 1 496
its TO entity (the reverse) -1 401

An association connecting the two entities in neither direction is
refused, not defaulted — a wrong cardinality exposes a list as a single
object and builds cleanly, so there is no downstream check to catch it.

TestApp carries the natural control: Mappings.Order_Customer appears in both
its definitions with a different MaxOccurs each time. Pinned as
TestFixtureCarriesTheDirectionControl.

Two proposal decisions corrected by measurement

  • Inherited attributes are IN, not refused: 398 of 3,697 exposed attributes
    (10.8%). The stored Attribute names the DECLARING entity, which
    DeclaringMemberRef already resolves for mappings.
  • Example is IN — see above.

Not guessed

Studio Pro pluralises a repeating element's exposed name. mxcli defaults to
the entity's own name and lets as 'Orders' say otherwise — reproducing English
inflection needs -y → -ies and an already-plural detector, and a name the
author writes beats one a heuristic guesses. Same conclusion as array-item
naming in #272's first half.

A bug the example caught

After a CREATE that resolves a folder, the cached hierarchy predates that
folder, so a later create or modify did not find the collection and wrote a
duplicate (CE0122). I checked whether it was pre-existing: JSON structures
do not hit it, but only because their create path happens to call
applyDocumentFolder first. The create branch now invalidates explicitly.

Scope

Import and export mappings, and nothing else — confirmed three ways:
generated/metamodel (11.6 snapshot) declares messageDefinition on exactly
those two types; modelsdk/gen (newer) declares the ref only in
importmappings/exportmappings/mappings, with businessevents and kafka
declaring none; and across the corpus + TestApp, of 24 collections the only
documents naming one are 8 import and 8 export mappings.

Authoring is modelsdk-only, as rules, menus and layouts are; legacy refuses with
the established message. Reading works on both engines.

Verification

  • Round trip against the real document, with controls: reverting the path rule
    reports 14 diffs, reverting the marker reports 1.
  • 6 executor tests with controls: reverting the direction gives
    MaxOccurs = 1, want -1; reverting the type map gives Long, want Integer.
  • 9 visitor tests, 2 codec round-trip tests.
  • mdl-examples/doctype-tests/40-message-definition-examples.mdl runs clean and
    builds at 0 errors.

Full suite, go vet, gofmt, make check-mdl (463 PASS) and
make check-skill-mdl are green.

Reviewer notes

  • Two new lexer tokens. MEMBER (only MEMBERS existed) and EXAMPLE.
    ANTLR prefers the longest match so members still lexes as MEMBERS, and
    both are added to identifierOrKeyword so attributes called Member or
    Example still parse. COLLECTION already accepted singular and plural.
  • TestNewRegistry_HandlerCountSnapshot and
    TestEveryShowAlternativeProducesAStatement both caught missing wiring during
    this work
    — they are doing their job. The equivalent guard does not exist for
    CREATE/ALTER/DROP; worth adding separately, since it will likely flag
    pre-existing gaps that should not ride along here.
  • The merge of main needed a hand-resolved findings/mdl-executor.jsonl:
    merge=union resurrected a record main had deleted. Second time
    (see fix(mappings): refuse to rewrite a SOAP-sourced mapping instead of dropping it #369); the merge commit explains it.

🤖 Generated with Claude Code

ako and others added 7 commits September 2, 2026 20:46
…llections

First slice of #272's implementation (proposal in
docs/11-proposals/PROPOSAL_authorable_message_definitions.md). Parses only —
nothing reaches storage yet.

A message definition is the source for 74 of the 327 mappings in the demo
corpus (22.6%), and was the only one of a mapping's four schema sources a
script could not create. Unlike an XML schema or a WSDL it holds nothing
external: every element names an entity, an attribute or an association.

Statements:

  create [or modify] message definition collection M.Name [folder '...'] ( ... )
  drop | describe message definition collection M.Name
  show message definition collections [in M]
  alter message definition collection M.Name add|drop|rename definition ...
  alter message definition M.Collection.Definition add|drop|set member ...

Two lexer notes, since a new keyword is how scripts break. MEMBER is new (only
MEMBERS existed); ANTLR prefers the longest match so `members` still lexes as
MEMBERS, and MEMBER is added to identifierOrKeyword so an attribute called
`Member` still parses. COLLECTION already accepted singular and plural, so
`show message definition collections` needs no new token.

An association member spells out its target entity (Assoc/Module.Entity, the
shape mappings already use). That is load-bearing rather than decorative: the
stored MaxOccurs tracks the DIRECTION of traversal and not the association's
type — measured, all 927 resolvable associations in the corpus are `Reference`
yet 526 store 1 and 401 store -1 — so naming the target is what makes the
direction explicit instead of something a reader has to work out.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second slice of #272. The document can now be written; nothing calls
it yet (the executor is next).

model.MessageDefinitionCollection gains the metadata a CREATE OR MODIFY must
preserve (Documentation/Excluded/ExportLevel) and its elements gain
OriginalName plus, on an association node, the TARGET entity — both needed to
rebuild or describe a node, since the stored MaxOccurs depends on the direction
of traversal and cannot be recovered from the association alone. The modelsdk
reader now carries all of them.

The codec writer fills in everything derived, which is nearly the whole
document. Measured across the corpus with no exceptions: MinOccurs 0, Nillable
true, IsDefaultType false, MaxLength/FractionDigits/TotalDigits -1, all four
text fields empty, ElementType from the node's kind, PrimitiveType Unknown for
an object, Path from the position in the tree, MaxOccurs -1 on a definition root
(56/56) and 1 on a value (3697/3697). The one it does NOT derive is an
association's MaxOccurs, which the executor must resolve from the traversal
direction.

Authoring is modelsdk-only, as rules, menus and layouts are: the legacy writer
has no serializer for the document and building one would duplicate a shape the
codec already gets right. Reading works on both engines, and the legacy backend
refuses with the established message.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ro document

ako/TestApp's OrderMessageDefinitions is hand-authored in Studio Pro rather
than shipped in a marketplace module, which makes it the right oracle for a
write path. Round-tripping it — read into the semantic model, re-encode,
compare against the STORED BYTES — found three things wrong, none of which the
corpus census had shown:

1. Path is a chain of ORIGINAL names, not exposed ones, and an ASSOCIATION
   contributes TWO segments — its own name, then the target entity's:

       Order|OrderLine_Order|OrderLine|Amount
       ^root ^association    ^entity   ^attribute

   Confirmed afterwards at 4,707 of 4,707 elements across TestApp and the nine
   demo apps.

2. The typed-array marker on MessageDefinitions and Children is 2; the codec
   defaults to 3. Measured at 37 definition lists and 575 child lists, all 2.

3. Every exposed element serializes Children even when empty — a leaf
   attribute stores the bare [2]. Same MandatoryLists rule as a rule document's
   Flows and a custom handler's ParameterMappings.

Example is now carried rather than hardcoded empty. The corpus said empty in
4,686 of 4,686, which is why the proposal scoped it out — but TestApp's
hand-authored definition sets one, so it is author-set and rare rather than
unused, and hardcoding it would silently drop the one that exists.

The fixture also happens to carry the control the direction rule needs: the
SAME association, Mappings.Order_Customer, appears in both definitions with a
different MaxOccurs each time (1 from Order, -1 from Customer). Nothing
synthetic would have been as convincing.

Both round-trip controls verified: reverting the path rule reports 14 diffs,
reverting the marker reports 1.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fourth slice of #272. The feature works end to end.

describe -> exec against ako/TestApp's hand-authored collection reproduces
Studio Pro's document exactly: the executor reports "Unchanged", meaning
canon.Reconcile found the rebuilt content semantically equal and elided the
write. mxbuild 11.14.0 reports 0 errors on a project holding an mxcli-authored
collection and an ALTERed one.

The executor's job is resolution, and two resolutions need the domain model.
Both were wrong first and were caught by the round trip:

  MaxOccurs on an association follows the DIRECTION of traversal, not the
  association's type. Holder is the FROM entity -> 1; holder is the TO entity
  -> -1. An association that connects the two entities in NEITHER direction is
  REFUSED rather than defaulted: a wrong cardinality exposes a list as a single
  object and builds cleanly, so refusing is the only honest answer.

  PrimitiveType is MAPPED, not passed through. Measured across 3,372 exposed
  attributes: Long -> Integer, AutoNumber -> Integer, Enumeration -> String,
  everything else identity. A pass-through gets 279 elements wrong, and the
  round trip caught it on TestApp's ProductId (a Long stored as Integer).

Inherited attributes resolve to the entity that DECLARES them, reusing
DeclaringMemberRef — 398 of 3,697 exposed attributes in the corpus are
inherited, and qualifying one against the entity that merely uses it is CE1613.

`example '...'` is new syntax, added because describe emitting nothing for it
made describe -> exec lossy. Rare (1 of 4,707 elements) but real, and silently
dropping an authored value is the failure this project keeps guarding against.

ALTER edits the stored document rather than rebuilding it, so definitions the
statement does not mention are never round-tripped through the describer.
Dropping or renaming a definition a mapping still references is refused, naming
the mappings.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fifth and final slice of #272.

mdl-examples/doctype-tests/40-message-definition-examples.mdl runs clean
end to end and mxbuild reports 0 errors, including an import mapping bound to
the authored definition. It also demonstrates the same association traversed
both ways, which is the property most likely to be got wrong.

Writing it caught one real bug: after a CREATE that resolves a folder, the
cached hierarchy predates that folder, so a later CREATE OR MODIFY did not find
the collection and wrote a DUPLICATE (CE0122). The update branch gets the
invalidation free from applyDocumentFolder; the create branch has to say it.
JSON structures avoid this only because their create path happens to call
applyDocumentFolder first.

The skill section moved to reference/message-definitions.md — the body crossed
the 700-line bound — leaving a pointer with the two facts a reader needs before
they follow it.

The proposal is marked implemented, with two scope decisions corrected against
measurement rather than left as written: inherited attributes are IN (398 of
3,697 exposed attributes, 10.8%), and Example is IN (author-set, rare, and
silently dropped if hardcoded). It also records the five derivations that only
a hand-authored document revealed.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolving findings/mdl-executor.jsonl by hand: `merge=union` resurrected a
record main had DELETED.

main removed a 2026-09-01 navigation finding (consolidated into
docs-wiki/bug-patterns/), and union never deletes — so the merged shard had 506
records where main has 504 plus this branch's one. Resolved as main's shard
verbatim plus the new record: 505.

Second time this has bitten (see the merge in #369). Union is right
for concurrent appends to a file of independent records, which is what
.gitattributes documents, and wrong the moment either side deletes or rewrites
— and `make check-findings` counts records without noticing they came back from
the dead, so the check is on line counts against main, not on the tool.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's build-and-test failed on PR #380:

    --- FAIL: TestMxCheck_DoctypeScripts/40-message-definition-examples.mdl/legacy
        Execution error: failed to create message definition collection:
        creating a message definition collection requires the modelsdk engine

The doctype round trip runs every example through exec + mx check on BOTH
engines, and message definition authoring is modelsdk-only by design — the
legacy writer has no serializer for the document, and building one would
duplicate a shape the codec already gets right, including a typed-array marker
of 2 and an empty-but-present Children list, neither of which is the codec's
default.

engineSkipList is the established mechanism for exactly this; menus, rules,
layouts, navigation profiles and languages all have an entry for the same
reason. This adds the sixth.

Worth recording why local verification missed it: the test is behind
//go:build integration, so `go test ./...` never compiles it. `make vet` now
vets the integration tag set (from the #319 fix), which type-checks but does
not RUN anything. Running it needs mxbuild, which is available locally —
`MX_BINARY=~/.mxcli/mxbuild/11.13.0/modeler/mx go test -tags integration
./mdl/executor/ -run TestMxCheck_DoctypeScripts/<file>` — and that is what
verified this fix: legacy SKIPs with the reason, modelsdk PASSes at 0 errors.

Refs #272

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ako
ako merged commit a90222c into main Sep 3, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

create json structure builds different element metadata than Studio Pro — array items named "DataItem" not "Datum", root MinOccurs 0 not 1

1 participant