Skip to content

softschema 0.9.0

Choose a tag to compare

View all tags
@jlevy jlevy released this 26 Oct 01:39
v0.9.0
8085986
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

softschema 0.9.0 is a minor release: the Python and TypeScript implementations now read
the same YAML the same way or refuse it in both, every validation result records which
checks actually ran, and callers can require that a check completed.

Until now, a document could read one way under uvx softschema and another under
npx softschema: a raw U+0085 became a space in Python and stayed a character in
TypeScript, 1_000 was a number in one and a string in the other, a block scalar at the
end of the text kept its final newline in one only. Each such shape now gets one verdict
in both runtimes, and a shape the two engines cannot agree on is refused by both with a
message naming it. Separately, a declared status still selects the validation mode,
but it never proved a check ran; the new per-layer execution field does. The
TypeScript package also moves to zod 4.6 and a patched fast-uri.

What's Changed

Breaking changes and compatibility

Newly rejected input. Both implementations now reject, as yaml_parse_error, any
raw character outside YAML 1.2 c-printable (C0 controls other than tab, line feed and
carriage return; DEL; the C1 controls; U+FFFE and U+FFFF), the line separators U+0085,
U+2028 and U+2029, and a carriage return that does not begin \r\n, anywhere in the
YAML they read: frontmatter or a pure-yaml document, comments and keys included, and
every compiled schema file. The Markdown body is not checked. TypeScript previously read
all of these as content; Python folded a raw U+0085 in a flow scalar to a space, kept
U+2028 and U+2029, and treated a lone carriage return as a line break.

Write such a character as a double-quoted escape ("\N", "\L", "\P", "\r", "\f",
"\v", "\0", "\xHH"), which both engines decode identically. repair writes the
escape for you inside a single-line scalar value. To find affected files before
upgrading:

rg -l '[\x{85}\x{2028}\x{2029}\x{FFFE}\x{FFFF}\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]|\r(?:[^\n]|$)'

Shapes the two engines read differently, now one verdict. Each of these was read by
one runtime and refused, or read differently, by the other. The CHANGELOG has the exact
rules and the fix for each.

Shape Now
A raw tab in a plain scalar or as block spacing Rejected in both; quote the value
A plain = The string = in both, as YAML 1.2 reads it
A flow or quoted continuation line not indented past its parent Rejected in both; indent it with spaces
A flow-mapping key whose : is past 1,024 characters or on a later line Rejected in both; the 1,024 limit counts UTF-16 code units in both
1_000, 0b101, -0x10, +0x10, -0o17, .5e3 and similar yaml_unsupported_scalar in both; write the core form or quote it
A :, ? or # glued to a flow scalar, such as [a:, b] or "a"#c Rejected in both; add the space or quote the scalar
Spaces wider than an explicit block-scalar indentation indicator Rejected in both
A mapping entry with content but no : Rejected in both
%YAML 1.1, 1.0, 1.3, or an unknown directive Rejected in both; only %YAML 1.2 and %TAG are accepted
The integer -0 (also -00) Rejected in both, per the spec’s negative-zero rule
--- followed by U+0085, U+001C–U+001F or U+FEFF A fence in neither
A surrogate pair written as two \u escapes The one character it encodes, in both

Python values change in one case. A block scalar that ends the YAML text with no
line break after it (the last key of a frontmatter block, or the end of a pure-yaml file
with no final newline) now ends with "\n" under clip (|, >) and keep (|+, >+)
chomping, as it does anywhere else and as TypeScript already read it. Write |- to keep
a value without its final line break.

Depth limit. TypeScript counts an empty collection toward the 64-collection limit
and reports the limit before an earlier duplicate key or a later parse error, as Python
does.

Messages and kinds. message text may change in a minor release, and these did;
kind is the matching key. Frontmatter-md line numbers now count from the top of the
file. A YAML parse failure, and a schema_invalid record whose schema the reader
refused, begins with the backticked file path. The preflight message names the escape
to write. Control characters in a filename are written as #xHHHH. repair without
--contract reports a fenceless document or a non-mapping root with validate’s kinds
(no_frontmatter, yaml_not_mapping), and compile --check reports an unreadable
committed sidecar as drift (exit 1) rather than a usage error (exit 2). repair adds the
record code yaml_escaped_character; a consumer matching repair codes exhaustively
should add it.

Result shape. Every structural and semantic layer record gains a required
execution. Python layer-result constructors take it as an explicit keyword-only
argument, and the TypeScript layer interfaces require the same field. A consumer that
validates serialized results against a closed schema must accept the new fields.
A report written before 0.9.0 has unknown execution; preserve the absence or
revalidate rather than filling it in.

Schemas both runtimes now refuse. Each of these turns an artifact bound to such a
schema invalid (ok false, validate exit 1, under every status, with or without
--require) until the schema is fixed. They affect hand-written or third-party schemas
only: schemas written by softschema compile carry the canonical $schema and absolute
ids and use none of the refused keywords.

Construct Was Now Write instead
Unresolvable $ref on a branch the payload never reaches Python accepted; TypeScript refused schema_invalid, reason: reference, structural not_run, in both Fix or remove the reference
$schema other than Draft 2020-12 (draft-07 and others) Python validated as 2020-12; TypeScript refused as compilation schema_invalid, reason: dialect Delete the line or use the 2020-12 URI
$recursiveRef, $recursiveAnchor Python ignored; TypeScript evaluated reason: dialect $dynamicRef, $dynamicAnchor
Legacy dependencies TypeScript enforced; Python ignored reason: dialect dependentSchemas or dependentRequired
OpenAPI nullable TypeScript honored; Python ignored reason: dialect type: [..., "null"]
A resource identity outside the identity grammar Each engine normalized it differently reason: resource_identity; compile --schema-id exits 2 https://host/path, file:///path, lowercase urn:<nid>:<name>, or another scheme:path, with no fragment

The identity grammar admits only spellings that neither Python’s urllib.parse nor
Ajv’s fast-uri rewrites. Ordinary-looking ids it refuses include https://example.com
with no path (write https://example.com/), an explicit default port, an uppercase
urn:uuid:, a trailing empty # as in http://example.com/schema# (common in
draft-07-era schemas), and s3:// or gs:// ids. The spec lists the rules, and
tests/vectors/identities.yaml has a case for each.

Compiled schemas. A Python- or TypeScript-compiled schema changes only where a model
string carries one of the newly rejected characters, a carriage return, a tab, or is
exactly =: such strings are now double-quoted with escapes, so the sidecar reads back
in both runtimes. schema_sha256 is computed over canonical JSON and is unchanged by the
escaping.

TypeScript compiled schemas change for some Zod constructs; recompile. Zod 4.5 and
4.6, with the compiler’s normalizations of their output, change the compiled schema and
schema_sha256 of models that use tuples, empty tuples, intersections of inline
objects, number- or int-keyed records, records with constrained keys, duplicate union
members, z.preprocess(), a second .multipleOf(), the zod-built patterns for
z.email(), z.emoji(), z.ulid(), z.cidrv6(), z.base64url() and positional
.includes(), or z.ipv4()/z.ipv6() followed by another format check. A
.default() over a transform, or an authored default or examples on one, now keeps
that value as its Pydantic mirror does. Several of these now compile to exactly their
Pydantic mirror’s schema. compile --check reports drift for an affected model;
recompile its committed schema.

Zod 4.6 semantics in TypeScript models. String .min(), .max() and .length()
count code points, as Pydantic does. An object-level .refine() or .superRefine() on a
strict object now runs when the input has an unknown key and reports beside the
unknown-key issue. z.iso.datetime() rejects minute-precision input such as
2020-01-01T06:15Z, which Pydantic accepts.

Dependency floors. The Python package requires frontmatter-format>=0.5.0; the
TypeScript package requires zod ^4.6.6.

The migration plan
covers downstream adoption of the execution fields.

New features

  • Per-layer execution evidence. structural.execution and semantic.execution are on
    every returned validation record, including CLI JSON and repair results. The public
    ValidationExecution type is not_run, completed, or errored; a completed check
    may accept or reject, so read ok and errors for its verdict. A missing or malformed
    schema reports a preparation failure without claiming the payload was evaluated, and
    both layers stay visible when a schema accepts and a model rejects.
  • Required checks. require= on Python validate_artifact, load_artifact and
    validate_values, a require option on their TypeScript counterparts, and a
    repeatable validate --require structural|semantic in both CLIs. A required layer that
    did not complete is not ok and gains a check_not_completed error carrying layer and
    execution: an otherwise valid result becomes invalid, strict reads raise or throw,
    and validate exits 1. An unreadable artifact keeps its input_error outcome and its
    exit 2. Without a requirement, results are unchanged.
  • Enforcement warnings. When the effective mode is enforced and structural
    validation did not complete, the result carries document-enforcement-via-model-only
    if semantic validation completed, or document-enforcement-not-applied otherwise.
  • repair escapes a raw rejected character. A U+0085, U+2028, U+2029 or control
    character (other than a carriage return) inside a single-line scalar value is rewritten
    as a double-quoted scalar carrying the YAML escape, with one yaml_escaped_character
    record per value. The decoded value is exactly what YAML 1.2 gives the raw text; no
    character is replaced by a guess at what was meant. A character in a key, a comment, or
    a block scalar stays a parse error for the author. Both implementations.
  • Faster reads, same results. Python parses a strict JSON document with json
    wherever ruamel would read it identically, so a large .json artifact validates many
    times faster, and parses YAML with ruamel’s optional C extension wherever libyaml reads
    exactly as the pure-Python parser (when ruamel.yaml.clib is installed and is the
    checked libyaml build); every other text goes to the pure parser, so values and errors
    do not change. yaml_parser_mode() and softschema doctor report which parser is
    active, and SOFTSCHEMA_PURE_YAML=1 turns the C parser off. The TypeScript reader is
    now linear in the width of a mapping: a 0.4 MB block mapping that took 6 s takes 0.4 s.

Fixes and hardening

Reading and repair:

  • compile writes schemas the portable reader accepts. Both writers emitted U+0085,
    U+2028 and U+2029 raw, so a model whose description, default or enum value carried one
    produced a sidecar every later read refused. Each writer now escapes them, compile
    checks its own output, and two older holes of the same kind are closed: a CRLF string
    in the Python writer and a tab or bare = in the TypeScript writer.
  • Conform never folds a raw U+0085. Python repair --contract … --schema … could
    fold the character to a space without a record and report valid; conform now runs
    the preflight first, in both runtimes.
  • repair’s quoting pass trims space and tab only, so a rejected character in a
    comment is no longer dropped in one runtime and refused in the other.
  • A raw rejected character right after a backslash in a double-quoted value is left
    for the author instead of being escaped into a different value.
  • The Python frontmatter reader splits lines on \n alone, as TypeScript does;
    str.splitlines() also broke on U+2028, U+2029, U+0085 and form feed.
  • Python reader crashes fixed: a collection used as a mapping key is now
    yaml_non_string_key, and %YAML 1.0 or 1.3 is a yaml_parse_error, not a
    traceback. An unexpected exception in the TypeScript YAML reader reaches the caller as
    itself rather than as a parse error.
  • A YAML parse failure names the artifact, in both profiles, both commands and both
    implementations, and a schema file is named once in its diagnostic.
  • repair reports a non-UTF-8 artifact the way validate does
    (artifact_invalid_utf8 at input_error).
  • Control characters in a filename never reach stderr raw, so a hostile filename
    cannot drive the terminal. The TypeScript preflight’s error path no longer
    materializes a large input to count lines.

Validation and compile:

  • Python repair_and_validate_artifact(..., model=...) uses the supplied model for the
    final verdict as well as conformance, so a model rejection returns invalid after
    repair.
  • TypeScript repair reports input_error for a missing or invalid-UTF-8 artifact
    without an inferred contract, matching Python.
  • TypeScript validateArtifact without a bound schema reports
    skipped_reason: inferred_via_model only when the call supplies a semanticModel, as
    Python does.
  • Python validate_values(schema=...) with a schema file that does not exist reports
    schema_missing with the path.
  • Python schema preparation resolves every reachable $ref before checking a
    payload    , as Ajv already did while compiling, so --require structural gives one
    answer in both runtimes.
  • TypeScript-compiled schemas match Pydantic again for nullable fields under zod 4.5
    and later, for .startsWith(), .endsWith() and .includes() under zod 4.6, for
    .catch() fields under zod 4.6, and for a .default() over a transform.
  • TypeScript compile rejects a default that canonical JSON cannot carry: a NaN or
    infinite default or example fails with canonical JSON requires finite numbers instead
    of compiling to default: null, as Python compile already did.
  • Checked enforcement accepts nested nullable references to explicitly closed models.
  • The TypeScript validate --require with an unknown layer reports
    softschema validate: invalid require: <value> (exit 2), as the Python CLI does.

Host isolation (Python):

  • compile no longer restyles the host process’s own YAML output. With
    frontmatter-format 0.4.0, a compile left its representers on ruamel’s shared
    SafeRepresenter, so every later YAML(typ="safe") dump in the same process wrote
    block-style mappings and |- literals. frontmatter-format 0.5.0 keeps them on a
    private subclass.
  • repair no longer changes how other ruamel emitters spell null. Its null
    representer now lives on a softschema-private subclass instead of ruamel’s shared
    RoundTripRepresenter.

Packaging: the sdist no longer ships AGENTS.md, which carries repository policy for
agents, not package documentation. The wheel and npm package never included it.

Security.

CI. bun audit runs as its own dependency-audit job, so a new upstream advisory no
longer skips the TypeScript typecheck, tests, build, publint and golden runs. The
typescript and cross-impl jobs no longer persist the checkout token and install with
--ignore-scripts.

Testing and release safety

The cross-runtime agreement is measured, not assumed. A generated sweep of 350,355
number-like plain scalars, 210,000 flow-collection scalars around the indicator
characters, and a 160,000-case fuzz of flow and block structures find no shape the two
runtimes read differently, and none that one accepts and the other refuses, outside one
loud, unchanged class: a block scalar without an indentation indicator whose leading
space-only lines are indented differently from its first content line (429 cases).
Write an indentation indicator, or trim those lines. A 4,686-case
writer-to-other-reader sweep finds no compiled schema one runtime writes and the other
reads differently.

Release checklist Phases 1–4 pass on the release tree: lint, 587 pytest plus 21 in the
slow tier, both C-first and with SOFTSCHEMA_PURE_YAML=1; 503 bun tests with the
coverage gate, bun audit clean, tsc/biome/publint; golden corpus 102 py / 97 node / 102
bun; cross-impl parity OK; compile --check with no drift; and Markdown format-check
clean. The wheel and the npm tarball were installed into clean environments and
exercised on both artifact profiles, the quickstart produced byte-identical output
across implementations, and the skill bootstrap created both mirrors from each installed
CLI.

Downstream integration was verified before release: Metaproc’s serializer integration
passed 96 tests against the validation-execution source, and downstream verification
surfaced the nested nullable-reference defect fixed above.

Full changelog: v0.8.1...v0.9.0

Assets 2
Loading