Add a first-class Markdown type system to MarkdownUtilities

[CraftBrewzMusic]

Note

Completed in ebe22d8. The implementation added storage-neutral Markdown records; portable type and rule assessment across frontmatter, body, and context; schema resource resolution; structured diagnostics and fix-its; filesystem adapters; and the md-utils types CLI. The completion run reported all 980 Swift tests passing. The formal design is recorded in mdtype RFC 0001.

Summary

Add a first-class Markdown type system to MarkdownUtilitiesCore and elevate the reusable parts of the existing rules engine from the md-utils executable target into the library.

Markdown types describe complete Markdown records, not only YAML frontmatter. A type can constrain:

• frontmatter through one or more JSON Schemas;
• Markdown-native body structure, including headings, sections, and heading hierarchy; and
• external record context, such as a logical path or, in the future, a persistence location.

This should be completed before building a future MarkdownUtilitiesServer, whose HTTP resources will be backed by typed Markdown records.

Motivation

The existing md-utils rules commands provide useful matching and validation infrastructure, but the model and validator currently live under the CLI target. Library clients cannot define rules, validate documents, or ask which rules apply without depending on CLI implementation details.

There is also no native concept of a Markdown type. Server and persistence layers will need to:

• validate a record as a named type;
• find all records conforming to a type;
• allow one record to conform to multiple types;
• distinguish hard conformance requirements from advisory recommendations;
• validate YAML frontmatter, Markdown-native body structure, and external record context;
• preserve logical path or storage context independently of the persistence backend; and
• report parse and validation failures, including invalid YAML, as structured diagnostics.

Types and rules are related but distinct:

• A type answers: "Does this Markdown record conform to Book?"
• A rule answers: "Which records does this policy apply to, and what requirements or recommendations should be checked?"

Types and rules may share predicate evaluation, parsing, diagnostics, and context models without becoming the same domain concept.

Markdown records and Markdown documents

The type system must make a clear conceptual distinction between a MarkdownRecord and a MarkdownDocument.

MarkdownDocument

A MarkdownDocument is the parsed representation of the Markdown content itself. It contains concepts encoded in the Markdown text, such as:

• parsed YAML frontmatter;
• Markdown body text; and
• a Markdown AST or derived body structure.

A document has no persistent identity, logical path, database table, object key, revision, or other storage context. It is a parsed interpretation and may not be constructible when the source contains invalid YAML.

MarkdownRecord

A MarkdownRecord is the stored or addressable resource containing:

• canonical Markdown text;
• optional stable identity and revision information; and
• external MarkdownRecordContext.

A record can exist even when its Markdown or YAML is invalid. It may originate from a .md file, a SQLite row, an object store, a Durable Object, or an in-memory test store.

Type assessment accepts a MarkdownRecord, analyzes its canonical content into a MarkdownDocument when possible, and evaluates:

• frontmatter constraints against the document;
• body constraints against the document; and
• context constraints against the record.

Invalid YAML must therefore become a structured record-assessment diagnostic rather than escaping only as a document initializer error.

Conceptually:

MarkdownRecord
├── canonical content
├── identity and revision
└── context
        │
        ▼ parse and analyze
MarkdownDocument
├── frontmatter
├── body
└── AST

Conformance semantics

Structural, non-exclusive conformance

Type conformance is structural and non-exclusive. A record may conform to Book, Document, and Publishable simultaneously. Asking for all Book records returns every record that conforms to Book, regardless of its other types.

JSON Schema can express nominal-like declarations as part of the structure. For example, a Book type may require a tags array containing the literal value books:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["title", "tags"],
  "properties": {
    "title": { "type": "string" },
    "tags": {
      "type": "array",
      "contains": { "const": "books" }
    }
  },
  "additionalProperties": true
}

The declaration remains structural: the tag is simply one required part of the record's shape.

Requirements and recommendations

Required constraints affect conformance. Recommended constraints produce advisory diagnostics but do not make an otherwise conforming record fail.

The initial public diagnostic severities should be:

• error for failed requirements and invalid record content; and
• advisory for failed recommendations.

Each requirement and recommendation should have a stable id so diagnostics and future migrations can identify the originating constraint.

Type-definition format

Users store project type definitions under:

.md-utils/types/

Type definitions may be written in YAML or JSON. Both formats decode into the same source model.

The definition envelope contains:

• md-utils-type-schema, identifying the md-utils type-definition schema version;
• name, the stable, case-sensitive type name;
• version, a nonempty type-contract version string; and
• the three conformance domains: frontmatter, body, and context.

Semantic Versioning is recommended for version but is not required or enforced. Values such as 1.2.0 and draft-3 are both valid. The value should remain an opaque string in the public API.

Example YAML definition:

md-utils-type-schema: "1"
name: Book
version: "1.0.0"

frontmatter:
  schemas:
    - ref: ../schemas/document.schema.json
    - inline:
        $schema: https://json-schema.org/draft/2020-12/schema
        type: object
        required: [title, tags]
        properties:
          title:
            type: string
          tags:
            type: array
            contains:
              const: books
        additionalProperties: true

body:
  requirements:
    - id: book-heading
      heading:
        text: Book
        level: 1

    - id: synopsis-hierarchy
      headingRelationship:
        parent:
          text: Book
        child:
          text: Synopsis
        relationship: directChild

  recommendations:
    - id: reviews-section
      section:
        heading:
          text: Reviews
        content: nonEmpty

context:
  requirements:
    - id: book-location
      path:
        glob: "books/**/*.md"

  recommendations: []

Frontmatter

frontmatter.schemas may contain one or more referenced or inline JSON Schema definitions. Every schema uses allOf semantics: the frontmatter must satisfy all listed schemas.

Frontmatter presence is derived as follows:

schemas is nonempty and presence is omitted → required
schemas is empty and presence is omitted    → not constrained
presence is explicitly optional             → validate when present; pass when absent

For example:

frontmatter:
  presence: optional
  schemas:
    - ref: ../schemas/document.schema.json

When presence is optional:

• present frontmatter must satisfy every listed schema;
• absent frontmatter passes this domain; and
• present but invalid YAML still fails assessment.

JSON Schema draft 2020-12 should be the initially supported draft. Schema reference resolution must not assume direct filesystem access in the pure assessment engine.

Body

body contains two subsections:

• requirements; and
• recommendations.

Both use the same extensible Markdown predicate vocabulary. Initial predicates should cover:

• heading presence, optionally at a specified level;
• heading relationships, including direct-child and descendant relationships;
• section presence;
• optional nonempty section content; and
• existing body limits that can be portably extracted from the current rules engine.

Heading and section analysis should use the Markdown AST rather than an ATX-heading-only line parser. Heading text should be compared against rendered plain text.

For hierarchy checks:

• directChild means the parent is the child heading's nearest lower-level ancestor;
• descendant means the parent occurs anywhere in the child's ancestor chain; and
• direct children need not be exactly one heading level deeper unless explicit levels are specified.

The model must remain extensible to links, wikilinks, section ordering, body limits, and other AST predicates.

Context

context describes where or how the canonical Markdown record is held. It is external to the Markdown document.

Initial support should include a normalized logical record path and glob predicates. A logical path must not require PathKit, FileManager, or a real filesystem.

Context should later be extensible to persistence-specific facts such as:

• a SQLite table;
• an object-store key or prefix;
• a Durable Object namespace; or
• another host-provided record attribute.

Persistence adapters translate native storage information into portable context values. The type checker itself must not open SQLite, inspect a filesystem, or contact an object store.

Path conditions may be used as candidate-selection optimizations, but path alone must not silently substitute for complete type assessment. A required path predicate is part of conformance; a query must still be based on successful assessment.

Normalized constraint model

Markdown-native constraints should compile into a typed, normalized predicate representation rather than custom JSON Schema keywords. JSON Schema validation remains one specialized predicate over frontmatter.

Types and rules can share concepts such as:

public enum MarkdownPredicate: Sendable {
  case frontmatterSchema(JSONSchemaResource)
  case heading(HeadingPredicate)
  case headingRelationship(HeadingRelationshipPredicate)
  case section(SectionPredicate)
  case path(PathPredicate)
  case body(BodyPredicate)
}

Type definitions and rule definitions remain separate models even when they compile into shared predicates.

Structured fix-its and md-utils types fix

Type diagnostics should be able to carry structured fix suggestions in addition to explanatory messages. A fix-it describes a proposed edit to the canonical Markdown record; it does not silently change whether the original record conforms.

Potential public concepts:

public struct MarkdownFixIt: Sendable {
  public var id: String
  public var title: String
  public var safety: MarkdownFixItSafety
  public var edits: [MarkdownRecordEdit]
}

public enum MarkdownFixItSafety: Sendable {
  case automatic
  case requiresInput
  case advisoryOnly
}

Fix-its should distinguish between deterministic edits and edits that need user input:

• a missing property with a JSON Schema const can offer the constant value;
• a missing property with a JSON Schema default can offer that value, but default remains an annotation and must never be applied silently;
• a missing property without a known value can request user input;
• a missing heading or section can offer a structural insertion when its location is unambiguous;
• a path mismatch can explain the required glob without moving a record automatically; and
• md-utils must never invent unknown domain values merely to make a record conform.

The portable library should produce fix-its and apply selected edits to an in-memory MarkdownRecord. Filesystem writes, interactive prompts, backups, and terminal presentation belong in MarkdownUtilities or md-utils.

The md-utils types command group must include a mutating fix command:

md-utils types fix Book books/dune.md
md-utils types fix Book books/ --dry-run

The command should:

1. assess each record against the requested type;
2. show the applicable fix-its and their safety classification;
3. request confirmation or required values interactively;
4. apply only the selected edits while preserving unaffected content;
5. write filesystem records atomically where supported; and
6. reassess the updated record and report its final conformance.

Required command behavior:

• --dry-run previews edits without writing;
• --yes accepts deterministic fixes but must not fabricate answers for requiresInput fixes;
• --constraint <id> limits changes to a particular constraint;
• failed recommendations are not changed unless --include-recommendations is explicit;
• types check and other assessment commands remain read-only; and
• a record that still does not conform after selected fixes produces a failing exit status.

Proposed library architecture

Portable assessment belongs in MarkdownUtilitiesCore. Native filesystem loading and scanning belong in MarkdownUtilities. ArgumentParser commands, terminal formatting, exit codes, and CLI orchestration remain in md-utils.

Potential public concepts:

public struct MarkdownRecord: Sendable {
  public var identity: MarkdownRecordIdentity?
  public var content: String
  public var context: MarkdownRecordContext
  public var revision: MarkdownRecordRevision?
}

public struct MarkdownDocument { /* parsed frontmatter + body + AST access */ }
public struct MarkdownTypeDefinition { /* name + version + compiled constraints */ }
public struct MarkdownTypeRegistry { /* validated definitions + lookup */ }
public struct MarkdownTypeChecker { /* record analysis + type assessment */ }

public struct MarkdownTypeAssessment: Sendable {
  public var type: MarkdownTypeName
  public var diagnostics: [MarkdownDiagnostic]
  public var conforms: Bool
}

public struct MarkdownDiagnostic: Sendable {
  public var code: String
  public var constraintID: String?
  public var fixIts: [MarkdownFixIt]
}

Type definitions should be decoded and compiled before record assessment. Invalid definitions, unresolved schema resources, duplicate active names, and unsupported md-utils-type-schema values are registry-loading failures, not record nonconformance.

Record analysis should parse canonical content once and reuse the resulting frontmatter, body AST, heading outline, and parse diagnostics when assessing multiple types.

Filesystem loading of .md-utils/types/ belongs in MarkdownUtilities. MarkdownUtilitiesCore should also allow callers to construct a registry from in-memory definitions and schema resources.

JSON Schema resolution

JSON Schema integration should be hidden behind a library adapter rather than exposing a particular validation dependency in the public API.

Initial reference behavior should:

• resolve relative references against the type-definition resource;
• support multiple inline and referenced schemas with allOf semantics;
• allow non-filesystem hosts to provide resources through an abstraction;
• detect missing resources, cycles, and conflicting schema identifiers;
• avoid implicit network access during pure assessment; and
• cache an immutable resolved schema graph for repeated assessment.

The current JSON Schema dependency must be validated for draft 2020-12 behavior, reference resolution, Linux, Swift concurrency, and WebAssembly compatibility before becoming a Core dependency.

Relationship to the current rules implementation

Preserve the existing separation between:

1. rule matching/applicability; and
2. rule checks/validation.

Conceptually:

Type:
record → requirements → conformance

Rule:
record → applicability predicates → applicable?
       → checks → policy diagnostics

Clarify that rules files-matching currently identifies files selected by a rule's match predicates; it does not necessarily mean those files successfully pass the rule's checks. Type queries must use successful type assessment, not rule applicability.

Rules should be able to reference named Markdown types, for example as applicability conditions, without making every schema-bearing rule a type. Type definitions must not depend on rules.

Reusable rule models, record analysis, predicate evaluation, validation, and structured results should move into a library target. Filesystem scanning, configuration loading, CLI formatting, and exit behavior remain separate adapters or orchestration.

Persistence considerations

The type engine must not assume that a record is an ordinary filesystem file. Canonical Markdown might come from:

• a .md file;
• a SQLite row;
• a Cloudflare Durable Object;
• R2 or another object store; or
• an in-memory or test store.

Path and storage information are record context rather than document content. Any extracted JSON representation, parsed document cache, search index, or type-membership index is derived state and must be rebuildable from canonical Markdown records.

Acceptance criteria

• MarkdownRecord and MarkdownDocument have distinct, documented responsibilities.
• Type assessment accepts canonical record content and reports invalid YAML as a structured diagnostic.
• Reusable rule models and validation logic live in a library target rather than only in md-utils.
• CLI rule commands use the library APIs without behavior regressions.
• Named Markdown type definitions can be loaded from .md-utils/types/.
• Type definitions may be YAML or JSON and decode to the same model.
• Definitions use md-utils-type-schema, a stable name, and an opaque version string for which SemVer is recommended but not required.
• The three conformance domains are frontmatter, body, and context.
• A type can reference or inline multiple frontmatter JSON Schemas with allOf semantics.
• Nonempty frontmatter.schemas requires frontmatter by default unless presence: optional is explicit.
• A type can express required and recommended Markdown-native body constraints.
• Heading presence, level, and hierarchy can be assessed from the Markdown AST.
• Section presence and nonempty section content are distinct constraints.
• A type can express required and recommended record-context constraints.
• Logical path assessment does not require filesystem access.
• Type assessment returns structured errors and advisory recommendations.
• Each constraint can expose a stable identifier in diagnostics.
• Diagnostics can expose structured fix-its with automatic, input-required, or advisory-only safety classifications.
• Fix generation never invents unknown values or silently treats JSON Schema default as validation behavior.
• Portable APIs can preview and apply selected edits to an in-memory MarkdownRecord without filesystem access.
• The required md-utils types fix command can preview, select, apply, and reassess fixes for filesystem records.
• types fix --dry-run performs no writes, and noninteractive operation never fabricates required user input.
• Filesystem fixes preserve unaffected content and use atomic writes where supported.
• One record can conform to multiple types.
• Type queries use successful conformance, not only rule matching or candidate selection.
• Pure assessment logic does not depend on ArgumentParser, terminal output, PathKit, or direct filesystem APIs.
• Tests cover structural overlap, nominal-like schema constraints, schema allOf, optional frontmatter, invalid YAML, heading hierarchy, advisory constraints, path-aware conformance, deterministic fix-its, input-required fix-its, dry runs, and post-fix reassessment.
• DocC documents records, documents, types, rules, applicability, conformance, requirements, recommendations, errors, advisories, and fix-its.

Open questions

• What exact JSON Schema should validate YAML and JSON type-definition files for md-utils-type-schema: "1"?
• Should type names or type versions support aliases during migrations?
• Which schema-resource URI schemes should the portable registry support initially?
• Should remote schema retrieval exist only as an explicitly supplied native adapter?
• How should logical paths be normalized across POSIX filesystems, object keys, SQLite records, and WebAssembly hosts?
• What initial vocabulary should represent storage context without coupling Core to particular persistence systems?
• Which existing rule matchers and checks belong in the portable predicate engine, and which require native context adapters?
• How should derived type-membership indexes identify the exact definition version or digest used to build them?

Dependencies

Blocked by:

• Split out a portable MarkdownUtilitiesCore library #75 — split out the portable MarkdownUtilitiesCore library so type and rule assessment have the correct target boundary.

Blocks:

• Ship the native Hummingbird 2 read-only server vertical slice #77 — MarkdownUtilitiesServer, which requires first-class Markdown types and reusable rule assessment.

This issue can proceed in parallel with #76 after #75 is complete. WebAssembly support and the type system are independent sibling prerequisites of the server.

Non-goals for this issue

• Implementing the HTTP server.
• Selecting SQLite, files, R2, or Durable Objects as the universal persistence backend.
• Defining every future storage-context predicate.
• Generating HTTP endpoints from OpenAPI.
• Treating cached JSON columns, parsed documents, or type-membership indexes as canonical data.
• Making type conformance nominal or exclusive.

[DandyLyons]

Implemented and pushed to main in commit ebe22d8: ebe22d8. This delivers storage-neutral Markdown records, portable type and rule assessment, frontmatter/body/context conformance, schema resource resolution, structured diagnostics and fix-its, filesystem adapters, the complete md-utils types CLI, mdtype filename conventions, and updated treedocs documentation. The full Swift suite passes with 980 tests.