Ion codec for Morphir IR: checkpoint 1, v3 document shape #938
Replies: 12 comments
|
Checkpoint 2. A specification may be written as several specs for one package, one module, or both. The split is an authoring group. The IR still stores one package specification and one module specification. A Ion-to-Ion collapse keeps the fragments and their The current note is the opening post of this discussion. |
|
Checkpoint 3. Specification annotations put the noun first: The package The opening post of this discussion is the current note. |
|
Checkpoint 4. A top-level The reader uses document order. An inline list is contributed where its struct appears, and a later top-level spec appends. A repeated type or value name in the merged module is rejected. A writer that starts from the IR emits the merged members inline. The opening post of this discussion is the current note. |
|
Checkpoint 5. A v3 dependency is a A writer omits an array that is allowed to be empty. A reader treats the missing array as empty. This covers The opening post of this discussion is the current note. |
|
Checkpoint 6. The The opening post of this discussion is the current note. |
|
Checkpoint 7. A member specification uses the same annotation slots as a definition.
The opening post of this discussion is the current note. |
|
Checkpoint 8. The opening post of this discussion is the current note. |
|
Checkpoint 9. The opening post of this discussion is the current note. |
|
Checkpoint 10. v3 values are locked. A definition is IR v4 is the next slice. The opening post of this discussion is the current note. |
|
v4 continues in #942. The v3 decisions in this thread stay as they are. |
Document tree. A v3 Ion document is one file. The datagram and the record in the opening post are that file. A document tree is a v4 storage profile. The CLI refuses to write v3 IR as a tree ( The same library as a v3 datagram. Names are canonical: After migration, the Ion document tree is a directory. The writer emits the JSON profile's canonical text. That text is one Ion value, and a number keeps its lexeme.
A reader also accepts the same members with symbol field names: The v3 spelling in the opening post stays as it is. The tree rules, value files, and dependency files are in #942. |
|
Document tree, revised. A tree file no longer holds a JSON-shaped value. Each file holds the same annotated elements as the single-file distribution. The path is the scope. The design applies to v3 and v4. This comment covers v3. v4 trees are in #942. The work is on branch The tree uses the same paths as the JSON and YAML trees, with the Directory segments and stems are the escaped file stems. For example, The rules:
An example for the package
The tree reads as this single-file datagram: Two cases that the reader refuses. The name does not match the path. This The path names the type The merge defines a type twice. This After the merge, The earlier document-tree comment is superseded. The v3 spelling in the opening post stays as it is. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Ion codec for Morphir IR
Checkpoint 1. This note decides the Ion document for IR v3, including values. IR v4 is the next slice.
The local copy of this note lives in
.dev/and is not committed. The shared copy is the finos/morphir discussion named in the checkpoint log below.Checkpoint log
package::specandmodule::spec. The name is the only group keymodules,types, andvaluesmerge with the matching top-level specspackage::specunder the same merge rules. Empty arrays are omittedionVersionin themorphirblock, SemVer, omitted means latestpublic::spec::typeAlias. Package and module stay grouping headersopaque,derived, andaliasare modifiers in front oftypecustomis a modifier in front oftypeas wellpublic::def::valueandpublic::spec::value. Bodies are S-expressionsDecisions
morphir_footer::{}ends the document.morphir::struct. Packages hold modules, and modules hold types and values. It is the collapsed datagram.public::def::alias::type. A member specification uses the same slots:public::spec::opaque::type.opaque,derived,alias, andcustomare modifiers in front oftype. A specification is the public face, so its access ispublic. Package and module stay grouping headers:package::specandmodule::spec.::. A Morphir annotation is the annotation list on a specification. v3 specifications in this model do not carry that list. The name is reserved so v4 does not reuse it.morphir/SDK:basics#intis a fully qualified name.basicsis a module name.intis a local name. A v3 path array is rebuilt from the canonical string.formatVersionis the release string3.0.0. v3 acceptskind: libraryonly.attributesfield. Writers omit it when the payload is the empty list. Readers treat a missing field as that empty list.package::specstructs with the samename, or severalmodule::specstructs with the same package andname. Apackage::specmay also carry optionalmodules. Amodule::specmay carry optionaltypesandvalues. The reader concatenates the inline lists with the matching top-level specs, in document order, into the one specification the IR stores. There is no separate group field. Adefmodule is not split. A repeateddefmodule name is rejected.dependencies,modules,types,values,typeParams,constructors,args, andarguments.ionVersionin themorphirblock is a SemVer 2.0 string and starts at0.1.0-draft.1. It is separate from IRformatVersion. A missingionVersionmeans the latest version this reader implements. Writers emit the field. Parsing uses thesemvercrate, not a hand-written version type.public::def::valueorprivate::def::value. A v3 value specification ispublic::spec::value. The body is an S-expression whose first symbol names the IR node. Apply is binary. A bareint,float,true,false, symbol,(), or[]is the compact form in the tables below. Writers omit an attribute struct when the payload is[].Why two layouts
The datagram matches a streaming read. The codec already walks a distribution as a header, then one dependency, then one module, then the end. Each top-level struct can become one of those events without holding the rest of the file.
The record is the same information nested as a tree. Tools that read one Ion value can load it. Collapse is the bridge from the datagram to that tree.
JSON and YAML do not need the record in order to exist. Either Ion layout is read into the IR, and the existing JSON and YAML writers emit their documents from that IR. Collapse does not itself produce JSON. What it does provide is one tree whose shape follows the IR, which is the shape those writers already walk. A transform that never builds the IR would still have to map Ion annotations onto JSON tags, and map value S-expressions onto the classic value arrays.
Annotation grammar
A definition and a member specification use the same order. The role is
deforspec.public,privatepublicdef,specmodule,value, or a modifier plustypealias::type,opaque::type,derived::type, andcustom::typereplace the single tokenstypeAlias,opaqueType,derivedType, andcustomType.opaque::typeandderived::typeoccur as specifications. A v3 definition ismodule,alias::type,custom::type, orvalue.A package or module specification is a grouping header. The noun comes first and there is no access annotation.
package,modulespecpublic::def::alias::type::is a public type-alias definition.public::spec::opaque::type::is an opaque type on a public face.public::spec::derived::type::is a derived type specification.package::spec::opens a dependency package.module::spec::opens a module specification. The header annotation ismorphir. The footer annotation ismorphir_footer. Those two are document markers and do not takedeforspec.Ion contract version
ionVersionversions the Ion spelling.formatVersionversions the IR inside it. The two fields stay separate. IRformatVersionkeeps the support table[3.0.0,3.1.0),[4.0.0,4.1.0).ionVersionfollows SemVer is the default contract versioning scheme.This contract is new, so it starts at
0.1.0-draft.1. A draft matches only that exact string. There is no released0.1.0yet, so the supported set is=0.1.0-draft.1. When a release exists, a reader accepts the current released major, the previous released major, and the exact drafts it lists. On0.y.zthe minor acts as the major, so0.1.xstays compatible and0.2.0is a break. A reader ignores a member it does not understand unlesscriticalnames it.A missing
ionVersionmeans the latest version that reader implements. Today that is0.1.0-draft.1. Writers still emit the field. A string that is not canonical SemVer, or a version outside the supported set, is rejected. The Rust reader parses it withsemver::Version.The same field sits on the record. The footer does not repeat it.
Reader dispatch
A datagram starts with
morphir::, continues with zero or more definition structs, and ends withmorphir_footer::{}. A footer is required. A truncated stream is invalid.A record is a single
morphir::value and has no footer. Further values after a record are invalid.v3 datagram
Writers emit the header, then each dependency package, then that package's modules, types, and values, then the distribution's own modules, types, and values, then the footer.
A missing
packagefield means the distribution package. A type or a value always carriesmodule. A reader rejects a type or value whose module was not declared. A repeated type name or value name in one module is rejected. A repeatedpackage::specname, or a repeatedmodule::specname in one package, is another part of the same specification and is accepted. List order inside the IR follows stream order.accessinsidecustom::typeis the constructor group. Thepublicannotation is the type's own access. v3 has both.docis omitted when the classic documented string is empty. A reader uses the empty string whendocis absent.v3 record
Collapse groups the datagram by package and module, drops
packageandmodule, and drops the footer.Expand is the reverse. Children of a dependency package receive
packageequal to that package name. Children of the distribution package omitpackage. Types and values receivemodule.v3 type expressions
A string that is a canonical fully qualified name is a reference with no arguments. A string that is a canonical local name is a variable. Any other string is rejected.
"a"orvariable::{ name: "a" }"morphir/SDK:basics#int"orreference::{ name: "morphir/SDK:list#list", arguments: ["a"] }tuple::[ "morphir/SDK:basics#int", "morphir/SDK:string#string" ]record::{ fields: [ { name: "score", type: "morphir/SDK:basics#int" } ] }extensibleRecord::{ variable: "r", fields: [ { name: "email", type: "morphir/SDK:string#string" } ] }function::{ parameterType: "morphir/SDK:basics#int", returnType: "morphir/SDK:string#string" }unit::{}A function type is binary. A list type is a reference to
morphir/SDK:list#list. A bare list is a tuple. Writers use the compact string when the node has no attributes and the compact string can say the node. Attributes, when present, force the expanded struct.alias::typecarriestypeParamsandtypeExp.opaque::typecarriestypeParams.derived::typecarriestypeParams,baseType,fromBaseType, andtoBaseType. The two conversions are canonical fully qualified names.custom::typecarriestypeParams,access, andconstructors.v3 type definitions are
alias::typeandcustom::type. v3 type specifications addopaque::typeandderived::type. A custom type on a public face ispublic::spec::custom::type.Spec groups
The IR stores one specification for a package and one specification for a module. The package
name, and for a module the package plus the modulename, are the only keys. There is nogroupfield.A top-level
package::specmay includemodules. Each entry is amodule::specand inherits that package name. Those entries combine with every top-levelmodule::specwhosepackageis that package and whosenamematches. A top-levelmodule::specmay includetypesandvalues. Those entries combine with every top-level type spec and value spec that names the same package and module. A nested type or value omitspackageandmodule.The reader walks the document from start to end. An inline list is contributed at the struct that holds it. A later top-level spec for the same name appends. An earlier top-level spec stays ahead of a later inline list. After that merge, a repeated type name or value name in one module is rejected.
Ion-to-Ion copy may keep the inline lists and the top-level specs side by side. A writer that starts from the IR emits one
package::specper package and onemodule::specper module, with the merged members inline, because the IR has already combined them.defdoes not use this merge. Onepublic::def::moduleowns the module's definitions.The reader builds one
morphir/SDKpackage and onebasicsmodule. Its types areint, thenbool, thenfloat.Dependencies
A v3 library carries its own package as definitions. The header
packageNamenames that package. Everypackage::specis a dependency, and so is everymodule::spec, type spec, or value spec whose package is not the header package. Apackage::specwhosenameequals the header package is rejected. Amodule::specwith nopackagefield belongs to the header package, so in v3 it is rejected too. The library's own modules arepublic::def::moduleand the otherdefstructs.Dependency order is the order of first mention. The first
package::spec,module::spec, or member spec that names a package opens that dependency. A later fragment for the same name appends. Inside a dependency, inlinemodulesand top-levelmodule::specvalues combine by module name, and inlinetypesandvaluescombine with top-level member specs, as in Spec groups.Empty
modules,types,values, andtypeParamsare omitted.The reader builds two dependencies,
morphir/SDKthenexample/ratings.basicshasint,bool, andfloat.listhas one opaque type.Scorehas one alias. No empty arrays are written.The same library as one record nests each dependency once. Collapse merges
basicsand still omits empty arrays.v3 values
A definition is
public::def::valueorprivate::def::value. It carriesinputTypes,outputType, andbody. A specification ispublic::spec::value. It carriesinputsandoutputand has no body. An input omits its attribute payload when that payload is[]. EmptyinputTypesand emptyinputsare omitted.Inline
valueson a module combine with top-level value structs for that module, in document order, by the same rule as types.A bare Ion
intis a whole number. A barefloatis a float.trueandfalseare bools. A bare symbol in expression position is a variable.()is unit.[]in pattern position is an empty list. A string, a character, a decimal, a reference, and a constructor keep a head, because a bare symbol is already a variable.score(ref 'morphir/SDK:basics#equal')(constructor 'example/finance:Eligibility#Ok')(apply fn arg)(lambda [score] body)(if cond then else)(let score { outputType: "morphir/SDK:basics#int", body: 70 } score)(letrec ((f { outputType: "...", body: (lambda [x] x) })) f)(match scrutinee [pattern body] [pattern body])(destructure pattern value body)(field record fieldName)(fieldFunction fieldName)(record (fieldName value) (fieldName value))(update record (fieldName value))(tuple a b)(list 1 2 3)()(string "hello")(char "a")(decimal "10.50")(lambda [score] body)is the compact pattern. It means anaspattern over a wildcard, with empty attributes. The explicit pattern is(lambda [(as (wildcard) score)] body). A writer uses the compact parameter when that is what the IR holds.A bare symbol in pattern position is that same compact binding.
_is a wildcard._scoreor(as (wildcard) score)(as pattern name)(tuple p1 p2)(constructor 'example/finance:Eligibility#Ok' p1)[](headTail head tail)()When a node has a non-empty attribute payload, a struct follows the head. Writers omit that struct when the payload is
[].A
letand aletrecbinding carry a value definition. EmptyinputTypeson that definition are omitted. Thebodyof the binding is an expression.Next checkpoint
IR v4 is discussion 942. The deferred nodes are below. v3 readers reject them.
Deferred to v4
specsandapplicationdistributions, entry points, structured attributes, holes, native and external and incomplete definition bodies, derived type definitions, and Morphir annotation arguments.All reactions