Ion codec for Morphir IR: v4

[DamianReeves]

Ion codec for Morphir IR, v4

This note continues discussion 938. The v3 rules stay in force: two layouts, the annotation DSL, spec grouping, empty-array omission, and expression S-expressions. ionVersion stays 0.1.0-draft.1 while this draft is refined in place. formatVersion is 4.0.0.

The local copy lives in .dev/ and is not committed.

Checkpoint log

Checkpoint	Scope	Discussion
1	v4 distributions, dependencies, entry points, and value bodies	#942
2	Structured attributes, hole expressions, incomplete types, Morphir annotations	#942
3	The v4 document literal, and the lexeme of a v4 float	#942

Decisions

1. formatVersion: "4.0.0" selects the v4 IR. A v3 reader rejects a v4-only node. A v4 reader accepts the v3 expression heads and the v4 nodes in this note.
2. kind is library, specs, or application.
3. A library carries its own package as definitions and its dependencies as specifications.
4. A specs distribution carries its own package and its dependencies as specifications. A module::spec with no package field belongs to the header package.
5. An application carries its own package and its dependencies as definitions. package::def groups those definition modules the same way package::spec groups specification modules. Inline modules combine with top-level modules for that package, in document order. A repeated type or value name in the merged module is rejected.
6. Entry points live on the morphir block of an application. The field is omitted for library and specs.
7. An expression definition stays public::def::value or private::def::value. The other v4 bodies are modifiers in front of value: native::value, external::value, and incomplete::value.
8. v4 inputTypes and inputs are structs keyed by parameter name. Field order is parameter order. A local name that is not an Ion identifier is a quoted symbol. outputType is required on an expression, native, and external body, and optional on an incomplete body.
9. v4 attributes are a struct. Type attributes may carry source, constraints, and extensions. Value attributes, including patterns, may carry source, inferredType, and extensions. Empty members are omitted. An empty attribute struct is omitted. On an S-expression the struct follows the head.
10. A hole expression is (hole reason) or (hole reason expectedType). The reason uses the same spellings as an incomplete body.
11. An incomplete type definition is public::def::incomplete::type or private::def::incomplete::type. It carries incompleteness and an optional partialTypeExp.
12. Morphir annotations are an optional annotations list on a specification. A definition rejects that field. A compact annotation is the string pkg:mod#local or pkg:mod#local:free text. A structured annotation is { name, arguments }. Arguments are value expressions, or { name, value } when named.
13. A v4 document literal is (document <payload>). The payload is the document itself. It is a tree of null, bool, string, number, list, and struct. A number with no point and no exponent is an Ion int. Any other number is an Ion decimal written with the stored lexeme. Ion float, timestamp, blob, clob, symbol, and s-expression are rejected inside the payload. A document is rejected in pattern position, and a v3 reader rejects the document head.
14. A v4 float literal that carries a lexeme is (float "<lexeme>"). A bare Ion float means the lexeme is the shortest spelling of that finite value. The lexeme is not part of the literal's identity.

Distributions

morphir::{
  ionVersion: "0.1.0-draft.1",
  formatVersion: "4.0.0",
  kind: library,
  packageName: "example/finance",
}

morphir::{
  ionVersion: "0.1.0-draft.1",
  formatVersion: "4.0.0",
  kind: specs,
  packageName: "example/finance",
}

morphir::{
  ionVersion: "0.1.0-draft.1",
  formatVersion: "4.0.0",
  kind: application,
  packageName: "example/finance",
  entryPoints: {
    serve: {
      target: "example/finance:App#main",
      kind: main,
      doc: "Start the service.",
    },
  },
}

An entry point name is the field name. target is a canonical fully qualified name. kind is the symbol main, command, handler, job, or policy. doc is omitted when absent. An empty entryPoints struct is omitted.

A specs distribution writes its own modules as module::spec and its members as public::spec::.... Dependency modules set package to the other package name.

module::spec::{
  name: "Eligibility",
  types: [
    public::spec::alias::type::{
      name: "Decision",
      typeExp: "morphir/SDK:basics#bool",
    },
  ],
}

package::spec::{
  name: "morphir/SDK",
}

module::spec::{
  package: "morphir/SDK",
  name: "basics",
  types: [
    public::spec::opaque::type::{ name: "int" },
  ],
}

An application dependency uses package::def. Its modules are access-controlled definitions. A top-level public::def::module or private::def::module with the same package combines with the inline list.

package::def::{
  name: "example/ratings",
  modules: [
    public::def::module::{
      name: "Score",
      types: [
        public::def::alias::type::{
          name: "Score",
          typeExp: "morphir/SDK:basics#int",
        },
      ],
    },
  ],
}

public::def::module::{
  package: "example/ratings",
  name: "Report",
}

The reader's dependency order is still the order of first mention.

Value bodies

inputTypes on a definition and inputs on a specification are structs. Order is the parameter order.

public::def::value::{
  module: "Eligibility",
  name: "approve",
  inputTypes: {
    score: "morphir/SDK:basics#int",
  },
  outputType: "morphir/SDK:basics#bool",
  body: (lambda [score]
          (apply
            (apply (ref 'morphir/SDK:basics#equal') score)
            70)),
}

public::spec::value::{
  name: "equal",
  inputs: {
    a: "morphir/SDK:basics#int",
    b: "morphir/SDK:basics#int",
  },
  output: "morphir/SDK:basics#bool",
}

A native body has no expression. hint is arithmetic, comparison, stringOp, or collectionOp. A platform hint is platformSpecific with a platform string. description is omitted when absent.

public::def::native::value::{
  module: "Basics",
  name: "add",
  inputTypes: {
    a: "morphir/SDK:basics#int",
    b: "morphir/SDK:basics#int",
  },
  outputType: "morphir/SDK:basics#int",
  hint: arithmetic,
}

public::def::native::value::{
  name: "now",
  outputType: "morphir/SDK:string#string",
  hint: platformSpecific::{ platform: "javascript" },
  description: "Current time on this platform.",
}

An external body has one binding per target platform. body is the optional fallback expression.

public::def::external::value::{
  name: "log",
  inputTypes: { message: "morphir/SDK:string#string" },
  outputType: unit::{},
  externals: [
    { targetPlatform: "javascript", externalName: "console.log" },
  ],
  body: (lambda [message] ()),
}

An incomplete body may omit outputType. incompleteness is draft, or hole with a reason. partialBody is omitted when the author left none. A hole reason is unresolvedReference with target, deletedDuringRefactor with txId, or typeMismatch with expected and found.

public::def::incomplete::value::{
  name: "draftScore",
  inputTypes: { score: "morphir/SDK:basics#int" },
  incompleteness: draft,
  partialBody: (lambda [score] score),
}

public::def::incomplete::value::{
  name: "broken",
  incompleteness: hole::{
    reason: unresolvedReference::{
      target: "example/finance:Missing#score",
    },
  },
}

The v3 expression heads stay valid. The v4 record-update node uses the same (update record (fieldName value)) head.

Attributes

source carries startLine, startColumn, endLine, and endColumn. All four are required when source is present. constraints and extensions are structs whose values are JSON-compatible Ion: null, bool, number, string, list, and struct. A symbol there is rejected, so the payload still fits the IR's JSON value. inferredType is a type expression.

variable::{
  name: "a",
  attributes: {
    source: { startLine: 2, startColumn: 10, endLine: 2, endColumn: 11 },
    constraints: { nonZero: true },
  },
}

(lambda {attributes: {
           source: { startLine: 4, startColumn: 1, endLine: 4, endColumn: 12 },
           inferredType: "morphir/SDK:basics#bool",
         }}
  [score]
  score)

A pattern uses the same value-attribute struct. (as {attributes: { inferredType: "morphir/SDK:basics#int" }} (wildcard) score) is a binding with an inferred type.

Hole expressions

A hole is a value expression. It is not a definition body. The reason is unresolvedReference, deletedDuringRefactor, or typeMismatch, with the same fields as an incomplete value body. expectedType is omitted when the hole has none.

(hole unresolvedReference::{ target: "example/finance:Missing#score" }
      "morphir/SDK:basics#int")

(hole {attributes: { source: { startLine: 8, startColumn: 3, endLine: 8, endColumn: 9 } }}
  typeMismatch::{ expected: "Int", found: "String" })

A v3 reader rejects hole.

Incomplete types

partialTypeExp is omitted when the author left none. typeParams is omitted when empty. The incompleteness spellings match the incomplete value body. Inside a hole incompleteness, partialBody is the type expression the author had written, and it is omitted when absent.

public::def::incomplete::type::{
  module: "Eligibility",
  name: "Widget",
  incompleteness: draft,
  partialTypeExp: record::{
    fields: [
      { name: "score", type: "morphir/SDK:basics#int" },
    ],
  },
}

public::def::incomplete::type::{
  name: "Broken",
  incompleteness: hole::{
    reason: typeMismatch::{ expected: "Int", found: "String" },
    partialBody: "morphir/SDK:basics#int",
  },
}

Morphir annotations

The field is annotations, written only when the list is non-empty. It may appear on module::spec, on a public::spec:: type, and on public::spec::value. A def node rejects it.

public::spec::alias::type::{
  name: "Score",
  typeExp: "morphir/SDK:basics#int",
  annotations: [
    "example/finance:Meta#deprecated",
    "example/finance:Meta#since:1.2",
    {
      name: "example/finance:Meta#range",
      arguments: [
        0,
        { name: "max", value: 100 },
      ],
    },
  ],
}

A positional argument is a value expression. A named argument is a struct with name and value. Empty arguments are omitted.

Document literal

Decision 0013 adds one v4 literal. Its type is the opaque SDK type morphir/SDK:document#document. The payload is the document verbatim, so the sexpr has one argument and that argument is the tree. { "value": 1 } inside the payload is a one-member document, not a wrapper around the document.

(document {
  name: "Alice",
  age: 30,
  tags: ["admin", "user"],
  metadata: null,
})

(document {
  id: 9007199254740993,
  ratio: 0.10,
})

age is an Ion int. id is an Ion int of arbitrary precision, so it does not pass through a 64-bit float. ratio is an Ion decimal written 0.10. The reader stores that lexeme. A number whose lexeme has a point or an exponent is never an Ion float.

The payload also allows a scalar. (document null), (document true), and (document "Alice") are documents. Struct field order is the object order. A duplicate field name is rejected. A field name that is not an Ion identifier is a quoted symbol.

These values are rejected inside the payload: an Ion float, a timestamp, a blob, a clob, a symbol used as a value, and an s-expression. Decision 0013 leaves out binary, timestamp, decimal-as-its-own-node, and reference extensions.

A literal pattern cannot hold a document. (document ...) in pattern position is rejected. A v3 reader rejects the document head, as it rejects hole.

Float lexemes

A v4 float literal stores the text it was written with. Two float literals are the same literal when they denote the same number. The lexeme is for writers that must reproduce the source.

(float "1.0e2")

A bare Ion float, such as 100.0, means the lexeme is the shortest spelling of that finite value. NaN, an infinity, and a magnitude no f64 can hold are rejected. A v3 float stays a bare Ion float. v3 stores an f64 and has no lexeme.

Next checkpoint

None. The v3 and v4 IR nodes now have an Ion spelling. Package-specification annotations stay out until issue 944 decides them.

[DamianReeves]

Checkpoint 2. v4 attributes are a struct: type nodes carry source, constraints, and extensions. Value nodes and patterns carry source, inferredType, and extensions. Empty members are omitted. On an S-expression the struct follows the head.

A hole expression is (hole reason) or (hole reason expectedType). An incomplete type is public::def::incomplete::type. Morphir annotations are an annotations list on specifications only. Compact annotations are strings. Structured ones carry value-expression arguments.

The opening post is the current note. The document literal is the next slice.

[DamianReeves]

Checkpoint 3. A v4 document literal is (document <payload>). The payload is the document itself: null, bool, string, list, struct, an Ion int for an integer lexeme, and an Ion decimal written with the stored lexeme for any other number. Ion float, timestamp, blob, clob, symbol, and s-expression are rejected inside it. A document cannot appear in a pattern, and a v3 reader rejects the head.

A v4 float literal that keeps its source text is (float "1.0e2"). A bare Ion float means the shortest spelling of that value. The lexeme is not part of identity.

That covers the remaining v4 literal. Package-specification annotations stay with issue 944.

The opening post is the current note.

[DamianReeves]

Superseded: the Ion document tree no longer uses JSON-shaped files. See #942 (comment).

Document tree. An Ion document tree is the same logical tree as the JSON and YAML profiles. The profile name is ion and the extension is .ion. Discovery reads manifest.ion. A second manifest in that directory (manifest.json, manifest.yaml, or manifest.yml) makes the root ambiguous.

Each file is one Ion value. Its members are the members of the JSON document-tree file at the same logical path: manifest, module, NAME.type, and NAME.value. The file has no morphir:: header, no morphir_footer, and no ionVersion. The extension selects the profile. The annotation DSL in the opening post stays on a single-file distribution.

A writer emits the JSON profile's canonical text: one object, spaces inside the braces, no trailing newline. A reader accepts that text. It also accepts Ion text whose field names are symbols. It reads an S-expression as a list and ignores annotations on a value. It rejects a blob, a clob, a timestamp, a symbol value, and a duplicate field name. A non-integer Ion decimal is read through Ion's display (1.5d2, 0.) and stored as a JSON number.

formatVersion in a tree file is the integer 4, the document-tree schema. The single-file header keeps the string "4.0.0".

Logical paths carry no extension. .type and .value are part of the logical name. The profile adds .ion at the physical boundary.

morphir-ir/
├── manifest.ion
├── pkg/
│   └── example/
│       └── finance/
│           └── eligibility/
│               ├── module.ion
│               └── decision.type.ion
└── deps/
    └── morphir/
        └── _sdk/
            └── @/
                └── basics/
                    ├── module.ion
                    └── int.type.ion

SDK is an initialism, so its directory is _sdk. The @ segment ends the package path. It stays bare while the model has no package version.

manifest.ion for a library. dependencies lists every package under deps/:

{ "formatVersion": 4, "distribution": "Library", "package": "example/finance", "pathBudget": 4000, "dependencies": ["morphir/SDK"] }

An application puts entryPoints on that manifest:

{ "formatVersion": 4, "distribution": "Application", "package": "example", "pathBudget": 4000, "dependencies": ["my-org/shared"], "entryPoints": { "start": { "target": "example:main#run", "kind": "main" } } }

pkg/example/finance/eligibility/module.ion names the node files. A public module omits access. A private module writes "access": "Private".

{ "formatVersion": 4, "path": "eligibility", "types": ["decision"], "values": [] }

decision.type.ion is a definition. Access is the key Public or Private. This matches the type-alias spelling MCK document-tree-0006 pins:

{ "formatVersion": 4, "name": "decision", "def": { "Public": { "TypeAliasDefinition": { "typeParams": [], "typeExp": "morphir/SDK:basics#bool" } } } }

deps/morphir/_sdk/@/basics/int.type.ion is a specification. A library dependency, and a specs distribution, carry spec. An application dependency carries def under deps/.

{ "formatVersion": 4, "name": "int", "spec": { "OpaqueTypeSpecification": {} } }

pkg/example/main/run.value.ion. The body stays the v4 JSON expression. The S-expression spelling from the single-file codec stays on that one file.

{ "formatVersion": 4, "name": "run", "def": { "Public": { "ExpressionBody": { "inputTypes": {}, "outputType": "morphir/SDK:basics#unit", "body": { "Unit": {} } } } } }

A hand-written type file with symbol field names reads as the same value:

{
  formatVersion: 4,
  name: "decision",
  def: {
    Public: {
      TypeAliasDefinition: {
        typeParams: [],
        typeExp: "morphir/SDK:basics#bool",
      },
    },
  },
}

A 4.0.0 tree has no member for a module specification's Morphir annotations list. Writing one fails with invalid_distribution_shape. The compatibility kit's profile list stays json and yaml. This profile is storage.

The v3 thread records the single-file side: #938. The draft section is docs/design/draft/ir/ion.md (“Document tree”). The file table is docs/spec/ir/schemas/v4/document-tree-files.md.

[DamianReeves]

Document tree, revised. This design replaces the earlier document-tree comment. Tree files no longer hold JSON-shaped values. Each file holds the same annotated elements as the single-file distribution. The path is the scope. The same rules apply to v3 (discussion 938). The work is on branch feat/amazon-ion-ir and is not merged yet.

Paths

The tree uses the same paths as the JSON and YAML trees. The extension is .ion.

• manifest.ion
• pkg/<package>/<module>/module.ion, <stem>.type.ion, and <stem>.value.ion
• the same three file kinds under deps/<package>/@/<module>/ for a dependency

Directory segments and stems are escaped file stems. morphir/SDK is deps/morphir/_sdk/@/.

The path supplies the package, the module, and the name of a node. A file may omit package, module, and name. A name that is present must match the path. The reader refuses a mismatch.

A root with manifest.ion and another manifest is ambiguous. The reader refuses it.

Manifest

manifest.ion holds the morphir:: header: ionVersion, formatVersion ("4.0.0"), kind, packageName, and pathBudget. It also holds one package::spec::{ name: "…" } for each dependency. That list keeps the dependency order. It also keeps a dependency that has no modules. A package::spec in the manifest may carry an inline modules list. Those modules merge with the deps/ files.

No tree file has a morphir_footer. The end of a file ends it.

Module, type, and value files

Under pkg/, module.ion holds exactly one public::def::module or private::def::module. The reader refuses a second one.

Under deps/ (library dependency), module.ion holds module::spec fragments. They may repeat, and they merge. The reader refuses a member that is stated twice.

module.ion may also hold the types and values of the module. They can be top-level elements or entries in the inline types and values lists of the module. A type or value inside module.ion states its name.

A type file holds exactly one type. A value file holds exactly one value.

Merge and order

The merge is additive, with the same rules as the single file. The module file comes first. The type files come next, and the value files come last. Each group is in path order. The reader refuses a type or value that the merge defines twice.

A tree orders modules and members by path.

Writer

The writer puts only the module header in module.ion. It writes one file for each type and one file for each value. It omits every name that the path supplies. The path budget can cut a stem to __ plus eight hex digits. Then the stem no longer spells the name, so the file keeps its name.

Example

A v4 library example/finance with one SDK dependency:

manifest.ion
deps/morphir/_sdk/@/basics/module.ion
deps/morphir/_sdk/@/basics/int.type.ion
pkg/example/finance/eligibility/module.ion
pkg/example/finance/eligibility/score.type.ion

manifest.ion:

morphir::{
  ionVersion: "0.1.0-draft.1",
  formatVersion: "4.0.0",
  kind: library,
  packageName: "example/finance",
  pathBudget: 4000,
}
package::spec::{
  name: "morphir/SDK",
}

deps/morphir/_sdk/@/basics/module.ion:

module::spec::{
}

deps/morphir/_sdk/@/basics/int.type.ion:

public::spec::opaque::type::{
}

pkg/example/finance/eligibility/module.ion:

public::def::module::{
  doc: "Credit eligibility.",
}

public::def::alias::type::{
  name: "decision",
  typeExp: "morphir/SDK:basics#int",
}

pkg/example/finance/eligibility/score.type.ion:

public::def::alias::type::{
  typeExp: "morphir/SDK:basics#int",
}

The tree reads as this datagram:

morphir::{ ionVersion: "0.1.0-draft.1", formatVersion: "4.0.0", kind: library, packageName: "example/finance" }
package::spec::{
  name: "morphir/SDK",
  modules: [
    module::spec::{
      name: "basics",
      types: [ public::spec::opaque::type::{ name: "int" } ],
    },
  ],
}
public::def::module::{
  name: "eligibility",
  doc: "Credit eligibility.",
  types: [
    public::def::alias::type::{ name: "decision", typeExp: "morphir/SDK:basics#int" },
    public::def::alias::type::{ name: "score", typeExp: "morphir/SDK:basics#int" },
  ],
}
morphir_footer::{}

decision comes before score because the module file comes before the type files.

A file may write a name that the path supplies, if the name matches. These files read the same as the files above.

deps/morphir/_sdk/@/basics/int.type.ion:

public::spec::opaque::type::{ name: "int" }

deps/morphir/_sdk/@/basics/module.ion:

module::spec::{ name: "basics", package: "morphir/SDK" }

The reader refuses this pkg/example/finance/eligibility/score.type.ion, because rating does not match the stem score:

public::def::alias::type::{
  name: "rating",
  typeExp: "morphir/SDK:basics#int",
}

Coverage

Coverage matches the single-file Ion codec. Today that codec reads and writes a v4 library. Specs and application distributions come later.