Skip to content

schema-calc v0.4.0

Choose a tag to compare

@fMindstormer fMindstormer released this 10 Nov 15:43
· 8 commits to main since this release

Invoice rules. A contract can now check every line of a list, use negative amounts and leave fields out, and join relations with and and or. With these, seven of the document-total rules of EN 16931, the European e-invoicing standard, compile into plain JSON Schema that gives the official Schematron's verdict on every one of 644 test invoices. Try it in any browser; nothing is sent to a server.

What's new

  • Constraints on each line. A constraint that reads lines.field outside sum() holds on every line: lines.amount = lines.net + lines.tax. It compiles once and is applied to every item, so its size does not depend on the list's length: on its own it is 20.5 kB for 4 lines or for 20. It reads one line's fields and constants only.
  • Signed amounts. "signed": true lets every amount carry one leading minus sign, as credit notes and allowances need. A signed schema is about four times larger: 3.6–4.7 times for the shipped examples.
  • Optional fields. "optional": ["paid"] lists fields that may be absent; in arithmetic an absent field counts as zero.
  • and, or, has and no. Relations can be joined, with and binding first, and has paid or no lines tests whether an optional field is present or a list has an item: has paid and due = total - paid or no paid and due = total. Up to eight relations and tests per constraint. A constraint that is one relation compiles exactly as before.
  • Smaller, shallower list sums. A list's sum now reads, in each column, how many lines have each digit, with contains, minContains and maxContains, instead of reading every line in turn. sum(lines) = subtotal at four integer digits is 16, 72, 240 and 682 kB for 1, 4, 10 and 20 lines, against 9, 72, 386 and 1,437 kB in 0.3.0, and a validator's decision path grows with the number of counts rather than lines. compile(spec, { listSums: 'positions' }) keeps the old construction.
  • Faster in Python. The root of every schema is now one reference, $ref: "_", to its body in $defs, so Python jsonschema crawls the document once per validation instead of once per constraint: the flat test corpus checks in 142 s instead of 493, and the larger test schemas 2.5 to 9 times faster. It costs 27 bytes: the invoice schema is 41,391 B.
  • Larger contracts. Up to 16 fields, from 8, and 4 lists, from 2.
  • The line-items example checks each line. lines.amount = lines.net + lines.tax and sum(lines.amount) = total: 16.66 + 3.33 = 19.99, 4.13 + 0.82 = 4.95 and 0.05 + 0.01 = 0.06, adding up to 25.00. Add one cent to a tax makes the first line 16.66 + 3.34, which the schema rejects although the total still adds up. It is 95,921 B.

EN 16931 totals in plain JSON Schema

A new page compiles BR-CO-10 to BR-CO-16 (line, allowance, charge and VAT sums; the totals without and with VAT; the amount due) and compares the result with the official Schematron, release validation-1.3.17. The two official syntaxes, UBL and CII, check some rules with different conditions on missing amounts, so each has its own contract, stating every rule as its Schematron checks it. On the 34 official examples and 610 copies with an amount raised by one cent or an optional element removed, all 644 verdicts agree, rule by rule, in Ajv and in Python jsonschema with its default options. Each schema is 9.6 MB, most of it for adding up 20 signed lines. Products, such as BR-CO-17's tax amount equal to taxable amount times rate, are not linear and remain out of reach.

Check it without Schema Calc

Python 3 and jsonschema only, in an empty directory:

python3 -m pip install jsonschema==4.25.1
base=https://github.com/fMindstormer/schema-calc/releases/download/v0.4.0
for f in validate.py invoice.schema.json invoice.valid.json invoice.invalid.json; do curl -fLO "$base/$f"; done
python3 validate.py invoice.schema.json invoice.valid.json invoice.invalid.json
ACCEPT  invoice.valid.json
REJECT  invoice.invalid.json

The line-items schema and records work the same way with lines.schema.json, lines.valid.json and lines.invalid.json.

Validator compatibility

The flat test corpus grew from 34,732 to 43,319 records across 85 schemas, adding constraints on each line, every one-digit signed sum, malformed signs, absent and explicit optional fields, signed lists, and relations joined by and and or against an independent oracle. All of them pass Ajv and Python jsonschema, and twelve validators in five languages ran them (table). None gave a wrong answer. Nine load every schema. @cfworker/json-schema needs --defs, as before: it refuses the 73 flat schemas that nest one embedded resource in another. @exodus/schemasafe times out loading 3 of the larger ones (1 with --defs). jsonschema-rs runs out of memory on 50 (46 with --defs), including all five shipped ones.

For library users

  • The root is one reference. A compiled schema is { $schema, $id, title, $ref: "_", $defs: { …, _: body } }. Code that read schema.properties or schema.allOf reads schema.$defs._ instead. Verdicts are unchanged.
  • and, or, has and no cannot name a field, a list or a line's field. A contract that used one of them as a name no longer compiles; the error names the field to rename.
  • Ajv's strict mode no longer warns about list sums, which no longer use partial prefixItems tuples. With listSums: 'positions' they do, as in 0.3.0.
  • Very large schemas need Ajv's resources added one at a time. On the 9.6 MB EN 16931 schemas, Ajv's own compile takes minutes and then overflows the stack while validating; prepareValidator, which the playground uses, compiles each resource on its own, bottom-up, and prepares them in well under a minute.
  • parseRule(source, spec) returns a constraint's clauses: parsed relations, and { has } or { no } tests. parseConstraint still parses one relation. normalizeSpec returns signed and optional when they are set.

Files

invoice.schema.json is the compiled invoice contract, 27 bytes larger than in 0.3.0 for the root reference, with the same verdicts; invoice.valid.json and invoice.invalid.json are the correct record and the one a cent off. validate.py is the standalone validator: it needs only jsonschema and never imports the compiler. schema-calc-0.4.0.zip is the source, with a MANIFEST.sha256 inside; schema-calc-0.4.0.zip.sha256 is its checksum and SHA256SUMS covers every file.

Scope

Sums and differences of decimal strings, signed if declared, their whole-number multiples and sums over lists, related by =, <=, >=, < or > and joined by and and or with presence tests: up to 16 fields, 4 lists of 20 items, 8 constraints and 12 digits. No products of fields, division, !=, parentheses, not, indexing into a list or other nested objects. The tally encoding takes no lists, signed amounts, optional fields or joined relations. A research prototype; it is not published to npm.