RFC: Bound the insight free-text fields (summary / detail / action) with maxLength

[peteski22]

Problem

schema/knowledge_unit.json declares the three insight fields as unbounded strings:

"summary": { "type": "string", "description": "What happened." },
"detail":  { "type": "string", "description": "Why it matters." },
"action":  { "type": "string", "description": "What to do." }

There is no maxLength anywhere on them, and neither SDK (sdk/python, sdk/go) constrains them either. That has two consequences for anyone building a conforming implementation:

1. Unbounded content meets bounded storage. A server that persists knowledge units in length-bounded storage (a fixed-width column such as varchar(n), a key-length-limited index, a size-capped document field) can accept a schema-valid unit at its API boundary and then fail, or be forced to truncate, at the storage layer. Truncation silently corrupts the insight (the caveat or the action can be in the dropped tail); failing turns a should-be-clean rejection into a low-level driver error.
2. Every implementation invents its own ceiling, and they drift. Because the schema sets no limit, each implementation picks one privately. A unit that is valid on one install becomes unstorable on another, and a conforming producer has no way to know what length is safe to emit. Cross-install portability of units is exactly what the commons depends on.

Today the only guidance is soft prose in the prompts (plugins/cq/skills/cq/SKILL.md: "summary — One-line description…"; sdk/*/prompts/reflect.md: "detail — two to four sentences…"). That prose shapes authoring, but it is not part of the contract and nothing enforces it: a producer that ignores it (or an LLM author that runs long) emits a unit that passes validation and breaks downstream.

Proposal

Add a hard maxLength to each insight field. Straw values, open to input:

Field	Role	Proposed maxLength	≈ words
summary	headline — "what happened"	500	~90
detail	substance — "why it matters"	8000	~1400
action	"what to do"	2000	~360

Rationale:

• These are ceilings, not targets. The soft prompt guidance keeps setting the target (summary is one line); the ceiling sits well above that so it only ever rejects pathological input, never normal authoring.
• summary is a headline, so a few hundred characters is generous. The fidelity of an insight lives in detail (and action), so those stay large; a tight summary cap does not cost fidelity because the substance was never meant to live there.
• detail carries the largest multiple deliberately. Its ceiling is sized against the machine author, not the soft target: real corpora already graze 4000 on a small first pass, and detail is the one field whose fat tail the "two to four sentences" guidance cannot bind. Fields with a bounded shape (summary is a headline, action is an instruction) stay near ~2× their observed maxima; detail is set well above that so an ordinary unit is never rejected as corpora grow.

The principle: hard where it must be enforced, soft where content is opaque

The fix should be consistent with how the schema already bounds things, which splits cleanly:

• Hard maxLength (schema-enforced) for fields that are core content, cross the wire, and land in bounded storage: summary / detail / action. A SHOULD cannot prevent a storage-layer failure, so these need enforcement.
• Soft SHOULD cap (documented, not enforced) for opaque, implementation-specific content that naturally lands in unbounded storage: the extensions slot. This matches the size-guidance direction already being scoped in Scoping the `extensions` slot (follow-up to #286) #411. extensions today is already the best-bounded part of the schema (maxProperties: 20, a namespace:key format constraint, producer SHOULD guidance); its values are intentionally opaque, so a soft cap is the right tool there. The insight fields are the conspicuous gap.

Related free-text fields (same class; bound together)

For completeness, these are also unbounded strings today and share the same "valid at the boundary, unstorable downstream" property. Proposing modest caps so we don't fix one field and leave the class latent; happy to split these into their own thread if they're contentious:

Field	Proposed maxLength
domains[] items	64
created_by	256
context.pattern	200
context.languages[] / context.frameworks[] items	64
flags[].detail	1000

Also bound array cardinality (maxItems)

The same "valid at the boundary, expensive downstream" property applies to the number of items in the producer-supplied arrays, not just each item's length. An unbounded domains/languages/frameworks array is schema-valid but can overrun bounded storage (a join table, an index, a size-capped document) the same way an unbounded string does. extensions is already cardinality-bounded (maxProperties: 20); these producer arrays are the remaining gap.

Adding a hard maxItems to the producer-supplied arrays (ceilings, not targets — nothing real approaches them):

Array	Proposed maxItems
domains[]	16
context.languages[]	16
context.frameworks[]	16

flags[] is intentionally left unbounded: it is server-managed and append-only over a unit's lifetime (not producer-supplied) and internal (never returned to querying agents), so a cap would block recording legitimate flags on a contentious unit rather than protect a producer.

Like the length bounds, these apply consistently wherever the arrays are redeclared at the wire boundary (propose.json, query.json), and are part of the same breaking tightening.

Compatibility and rollout

• This is a breaking tightening of the JSON Schema: units above the new ceilings become invalid. In practice essentially all existing units sit far below these bounds (the soft guidance already keeps them short), so real-world breakage should be minimal, but it should be flagged as breaking and reflected in how the schema version is tracked. No corpus rewrite is implied.
• SDKs: mirror the bounds in the Python models (Field(max_length=…)) and the Go types, with fixtures/round-trip tests.
• Prompts: keep the soft targets; add a one-line note of the enforced ceiling so authors know the hard limit.
• Implementer guidance (docs): size storage at or above the schema ceilings; enforce the schema's number, never a private, smaller one; never truncate — reject the whole unit with a clear error.
  • Make the rejection actionable: name the field, its limit, and the actual length. The producer is usually an LLM, and an error like detail: 8420 exceeds maxLength 8000 lets it re-emit a shorter unit on its own; an opaque validation failure just loses the insight.
  • maxLength counts Unicode code points, not bytes. Sizing a varchar(n) or byte-capped column needs up to 4× the number in bytes for UTF-8; mixed-script corpora make this a real source of storage-layer failure, not a hypothetical one.
• SDK constants: publish the ceilings as machine-readable constants in the SDKs, not only in the JSON Schema, so an implementer wanting a pre-flight check reads a provided number instead of hardcoding a private one.

Call for input

• @CaseyRo — Stolperstein Stolperfalle runs against schema/knowledge_unit.json and strips non-conforming fields at the wire, and you raised the extensions size-cap question in Scoping the `extensions` slot (follow-up to #286) #411, so your read here would be especially useful: do the straw ceilings fit what your deployments already store, and does the "hard for core insight fields / soft for extensions" split match how you'd want size bounded?
• Anyone maintaining an independent implementation: please flag if any proposed ceiling is below what your installs already persist, so we set the numbers high enough to avoid gratuitous breakage.

Builds on the extensions-slot scoping in #411 and the original schema-extensions thread in #286.

[CaseyRo]

Numbers from the Stolperfalle install, since you asked for real corpus data.

Corpus: 194 units, 2026-04-20 → 2026-07-29, one install. 150 pitfall / 31 workaround / 13 tool-recommendation. Worth stating up front: these are machine-authored — Claude-family models via a reflect extraction path plus session hooks, not humans typing into a form. That matters for where you set ceilings, see below.

Lengths in characters:

field	min	p50	p90	p95	p99	max	proposed cap	verdict
summary	70	180	229	241	267	280	500	✅ comfortable
detail	305	676	1391	1847	2657	3913	4000	⚠️ too tight
action	102	277	446	562	848	983	2000	✅ comfortable

Counts over thresholds: 60/194 summaries exceed 200 chars (0 exceed 300); 9 details exceed 2000, 1 exceeds 3000, 0 exceed 4000; 0 actions exceed 1000.

summary: 500 — yes. Our max is 280 with 1.8× headroom. Note the 60/194 over 200 chars, though: if anyone argues the headline cap down to 200 "because it's one line," that would invalidate a third of our corpus. 500 is the right kind of generous.

action: 2000 — yes. Max 983, ~2× headroom.

detail: 4000 — please raise it. 8000, or 6000 at minimum. One install, three months, 194 units, and we're already at 3913. A ceiling that a small corpus grazes on its first pass isn't a ceiling that only rejects pathological input — it's one that will start rejecting ordinary units as soon as somebody's corpus is 10× ours.

The reason is structural, and it's the thing I'd most want folded into the RFC: detail is the field the soft guidance fails to bind. "Two to four sentences" reliably shapes a human. It does not reliably shape an LLM, which will happily write four sentences that are each 200 characters and include a code block. Since most units in any real deployment are machine-authored, the ceilings have to be sized against the machine author, not the documented target. On the fields with a bounded shape (summary is a headline, action is an instruction) that gap stays small — our maxima sit at 56% and 49% of the proposed caps. On detail, which is where you correctly say the fidelity lives, the tail is fat and the guidance does not hold it. That's exactly the field that needs the multiple, not the tight fit.

Rule of thumb I'd suggest for the numbers generally: ceiling ≥ 2× the observed max on any field carrying substance. summary and action already satisfy that; detail at 4000 is at 1.02×.

The related fields — all fine against our data, no objection to any of them:

field	our max	proposed
domains[] items	24	64	✅
context.pattern	24	200	✅
context.languages[] / frameworks[] items	10 / 17	64	✅
created_by	56	256	✅

One note on created_by: ours is a did:key Ed25519 identifier, which is a fixed 56 characters — so our max tells you nothing about the real tail. did:web identifiers embed a domain and path and run much longer. 256 is the right call; I'd resist anyone tightening it toward what a did:key corpus suggests. No data to offer on flags[].detail — we don't persist flag detail, we map the reason to the upstream enum at the wire.

On the split — agreed, and for the reason you give. Hard enforcement where content crosses the wire into bounded storage, soft SHOULD where values are opaque and land unbounded. The extensions slot is genuinely the well-bounded part of the schema already (maxProperties, the namespace:key format constraint); its values are opaque by design and a hard cap there would be enforcement without a failure mode to prevent. The insight fields are the real gap.

Two additions to the implementer guidance, both cheap:

1. Make the rejection actionable. "Never truncate, reject the whole unit" is right, but the error should name the field, the limit, and the actual length. The producer is usually an LLM, and an LLM that gets back detail: 4780 exceeds maxLength 4000 will re-emit a shorter unit on its own. ValidationError alone just becomes an opaque failure and the insight is lost.
2. Say that maxLength counts Unicode code points, not bytes. Implementers sizing a varchar(n) or a byte-capped column need up to 4× the number in bytes for UTF-8. Our corpus is mixed German/English, so this is not hypothetical — the ö/ä/ü alone put a real gap between the two counts, and it's exactly the kind of thing that turns "schema-valid" into a storage-layer error you were trying to eliminate.

And one on rollout: publishing the bounds as machine-readable constants in the SDKs (not just in the JSON Schema) is what actually stops implementations from re-inventing private ceilings. We read the schema, but a Go or Python implementer who wants a pre-flight check before the wire will hardcode a number unless the SDK hands them one.

[peteski22]

Thanks — this is exactly the corpus data the RFC needed, and framing the ceilings against the machine author rather than the documented target is the right lens.

Changes made:

• detail → 8000. 4000 was at 1.02× your observed max; 8000 clears the 2×-observed-max rule and sizes against the LLM tail the soft guidance can't bind.
• Both implementer-guidance points are folded into the RFC: rejections must name field + limit + actual length, and maxLength counts Unicode code points, not bytes (size storage up to 4× for UTF-8).
• summary 500 and action 2000 stay (your headroom numbers confirm them); related fields and maxItems unchanged.

Schema + Go/Python SDK constants are up in #508; the actionable-error and code-points guidance lands with the SDK/server enforcement and docs (#503–#507).