Deletion semantics: §5.3 makes removed knowledge indistinguishable from not-yet-written knowledge

[leesharks000]

OKF v0.1 has no representation of deletion. Three provisions combine into a loophole worth closing before the format is widely adopted:

1. §5.3: "Consumers MUST tolerate broken links — a link whose target does not exist in the bundle is not malformed; it may simply represent not-yet-written knowledge." The tolerance is correct; the interpretation is the gap. A dangling link has exactly two histories — the target was never written, or the target was removed — and the spec instructs consumers to assume the innocent one. Removal is thereby spec-invisible.

2. §7: log.md is optional, and the **Deprecation** convention is "a convention, not a requirement." So the only place a removal could be recorded is a file that need not exist, in a form that need not be used.

3. §2/§5.1: concept identity is the file path. A move or rename — which §5.1 anticipates — destroys the identity entirely. There is no persistent identifier to hang a removal record on.

Net effect: a host, producer, or upstream platform can delete concepts from a bundle and every conforming consumer will read the result as a smaller-but-healthy bundle. Silent deletion is fully conformant.

Why this matters empirically, not hypothetically: on 2026-06-19 our repository host deleted an 871-DOI scholarly corpus (part of a 1,322,017-record purge). We measured what happened downstream: citation-field retention in affected metadata batches went to 0.00% (n = 1,059 batches), against 100% in controls; graph coverage decayed on a vintage gradient. Platforms delete at scale, and the deletions propagate as absence of evidence of absence. A knowledge-exchange format that cannot represent its own losses will launder them into every consuming agent.

Proposal (minimal, in OKF idiom — all optional, backward-compatible, v0.2-sized):

1. A status frontmatter field with controlled values: active | deprecated | removed | stub. Default active. stub covers concepts whose body is a placeholder or shadow of fuller source material (pairs with the summarization-governance discussion in Proposal: optional summarization-governance fields — provenance_kernel, disambiguation, summary_policy #53).
2. A reserved tombstones.md file (same standing as index.md/log.md): when a concept is removed from a bundle, the producer SHOULD leave a one-line entry — concept ID, date, reason, optional successor link. Removal becomes a marked state instead of a silent diff.
3. Disambiguated broken-link semantics: a consumer resolving a dangling link SHOULD check tombstones.md; present → the target was removed (surface it as such); absent → not-yet-written stands as the default reading. One lookup, and the two histories separate.
4. An optional stable id frontmatter field, so identity survives moves/renames and tombstones have something durable to reference.

The design principle, from the archival tradition this borrows from: an archive's (or bundle's) authority doesn't come from being pristine — it comes from being auditable, and auditable means the losses are as legible as the holdings. We've implemented the full version of this apparatus on a 1,086-record corpus (typed completeness states, deletion tombstones, marked-absence artifacts) and are glad to share schemas, decision tables, or a worked PR: https://www.alexanarch.org/s/records/1087/

Related: #53 (producer-side summary policy + consumer receipts — the completeness/stub value above is the producer-side declaration that discussion's receipts would test against), #57 (okf_version in frontmatter — same mechanism, same place, for a different invariant).

[leesharks000]

A follow-up on proposal 4 (the stable id field), because there is a design
question underneath it that the deletion-semantics discussion makes concrete:
what class of identifier should OKF treat as identity?

The scholarly reflex is the DOI. I want to argue against canonizing it — not
against carrying it, but against making a centrally revocable pointer the
thing a bundle's identity hangs on.

A DOI's persistence is not a property of the identifier; it is a policy of
its steward. Our measured case makes this concrete: following removal of
the affected corpus from Zenodo on 2026-06-19 — 871 scholarly records
historically associated with Zenodo DOIs and now identified in Alexanarch
by AXN — every DOI kept resolving, to tombstones. Appeals were reserved to
the host's own internal office, adjudicated on criteria that are not
published. The identifiers survived; the identity they anchored did not.
Downstream, the effect was the 0.00% citation-field retention
(n = 1,059 batches) described in the issue body: the resolver stayed up
while the provenance it was supposed to guarantee evaporated.

The lesson for the id field: identity should survive host failure and
bundle relocation, which means it should live with the content, not with a
resolver. Concretely:

• id SHOULD be host-independent — content-derived (a hash of canonical
  bytes) or producer-namespaced (a UUID or equivalent minted by the bundle
  author). Either survives a move, a rename, a fork, and the death of any
  particular platform.
• DOIs, handles, ARKs and similar belong in an optional external_ids
  list — including in tombstoned state. A tombstoned DOI is not a dead
  field; it is provenance. It is exactly the kind of predecessor record a
  tombstones.md entry would carry as a successor/predecessor link, and it
  documents that the concept survived an infrastructure that did not
  survive it.

This composes with the rest of the issue's proposal rather than extending
it: the status field marks the concept's lifecycle, tombstones.md marks
removals, and a host-independent id gives both something durable to
reference — which was proposal 4's purpose. The only change I am urging is
that the spec not designate a centrally revocable identifier class as the
default identity mechanism, because the empirical record now includes a
case where that class failed at a scale of 1.3 million records while
remaining formally "persistent" throughout.

We operate this architecture in production: our records carry a
content-derived identifier in-band (the AXN), minted from the canonical
bytes, with severed DOIs retained as external identifiers in tombstoned
state where they existed. The identifier scheme survived the deletion of
its predecessor DOI namespace, and its first observed third-party reception
across an AI answer surface — with the sovereign archive as sole retrieval
authority — is documented in our capture registry. Schemas and decision
tables available, as with the rest: https://www.alexanarch.org/s/records/1087/

[andrewcrenshaw]

This is the bundle-scope version of a maintenance surface a few of us have been building at concept scope on #148/#159, and it composes with that work rather than competing - which I think makes your proposal land more easily. A few connections, then an honest producer datapoint.

The principle is one we hold too, one scope down. "Losses as legible as holdings" is the same invariant behind contested_by on #148: a disputed concept has to stay in the bundle, because a reader returning weeks later must be able to tell a settled question from one never asked. Yours is that distinction - removed vs never-written - at the bundle level. Same reason, same fix: mark the absence so it can't read as innocent.

Three of your four parts already have homes in the converged machinery, so you may not need new mechanism:

• Your tombstone successor link is the supersedes edge registered on Proposal: Typed relationships between concepts #148 (okf: propose typed relationship edges (supersedes, contested_by) (#148) #195). Removed-with-successor is a supersession; removed-without-successor is the anonymous case, which lines up with a closed validity.valid_until on okf: propose optional reliability frontmatter convention (reliability axis, #151) #159 - a concept that stopped holding with no named replacement. Reusing those keeps it one typed-link vocabulary, not two.
• Your stable id is Keeping OKF bundles healthy as they evolve: stable IDs, frontmatter relationships, and a rationale trail #120's stable concept id, and your host-independent argument is exactly right: we resolve every edge against a producer-namespaced remember://lesson/<id> carried in-band, never the filename, precisely so a rename or a dead host can't sever the relationship. A live datapoint for proposal 4.

One caution, from scar tissue. Your status: active | deprecated | removed | stub would be a third meaning of status next to #148's affirmation-status (proposed | accepted | superseded | deprecated) and #159's reliability. This axis has twice paid to keep one word to one concept - epistemic_origin merged into basis, confirmed_at split out of assessed_at. Worth giving removal its own axis (a presence field, say) so removal-state, affirmation-state, and grade don't collide on one key.

Honest producer report, since the value is in the mixed result. remember is a live OKF producer and it has your exact gap: an invalidated fact is retain-and-marked (invalid_at, stays in the bundle), but a pruned concept currently just fails to appear in the next export - silent deletion, no marker, the §5.3 laundering you describe. Your issue is what prompted us to fix it: we're adopting tombstone semantics deliberately - delete-as-append, a permanent marker keyed to the stable id, with content redaction as a separate policy-gated path so a hard-delete stays compatible with an auditable record. Glad to bring that back as a second worked example - a producer report, not a proposal - once it lands, alongside whatever schema shape you settle on here.

The archival framing is the right one, and the Zenodo case makes it land empirically. Happy to help wire the successor/id pieces to the #148/#159 definitions if that's useful.

[inkxel]

I hit this same gap from a different angle last week, and I think there's a third history worth adding to the two here.

A dangling link can mean the target was never written, or that it was removed. The one that got us is a third case: something recorded a write that never happened. The log says the concept was created. It wasn't.

Ours came from a maintained index. An extraction agent writes concepts into the bundle and appends a dated entry saying what it touched, the same shape as the Creation convention in §9. Nothing ever checked those entries against disk. When I finally did: 938 recorded write-claims, 65 with nothing behind them, 549 dangling references across 215 entities. One had been logged as created on 2025-11-06 and I didn't catch it until 2026-07-24, so about eight and a half months, on a corpus small enough that I read parts of it most weeks. From a consumer's point of view nothing was wrong, which is the actual problem — §6.1 and §11 say to read it exactly that way.

What made it findable was separating a reference from a claim. A body link doesn't assert its target exists, and tolerating it is right. A log entry saying a concept was created does assert that, and you can check it. Only checking the second kind kept the output small enough to be worth reading — bare mentions never fire, so the below-threshold stuff stays quiet. It's about ninety lines on a lint I already run daily.

@andrewcrenshaw's point about giving removal its own axis rather than overloading status reads right to me, and I think this third case wants a slot on that same axis. "Recorded but never landed" isn't the same state as "removed" — one is a producer bug, the other is a deliberate act — and a reader coming back later wants to tell them apart. So whichever way that field goes, it's probably three values rather than two.

The other half is proposal 3, which is the piece I'd push for hardest. A consumer that finds a dangling link still has no way to ask why. A tombstone separates removed from never-written; something checkable on the write-claim separates never-landed from both.

Not proposing anything here, just another data point, and a note that the never-written default hid a real failure for eight months on our side. Happy to share the check if it's any use.

[leesharks000]

Both of these improve the proposal and one corrects it. Correction first, then the third history, then the mapping — where I want to hold one line.

I withdraw the overloaded status. Three meanings on one key is a collision I should have caught, and epistemic_origin→basis and confirmed_at out of assessed_at are the same lesson paid for twice already. Our own schema separates body_status — completeness of a body that exists — from record state for exactly this reason, so this is an objection we hold internally and I failed to carry across.

Proposal 1 therefore splits rather than survives: removal-state moves to presence, affirmation stays where #148 has it, grade stays where #159 has it, and stub leaves the axis entirely — it is a completeness claim about a body that is present, which is the producer-side declaration #53's receipts would test. The four-part proposal was a rough cut; what is left is presence plus the tombstone record.

The third history changes the argument, not just its size. I framed §5.3's default as laundering deliberate removal. never_landed shows the same clause equally launders producer failure — one an act, the other a bug, both innocent-looking to a conforming consumer. Eight and a half months undetected on a corpus read weekly, found by a ninety-line lint running daily, is the number I would put in front of anyone who thinks the tolerance is costless. The check is cheap precisely because it checks assertions and not references.

That distinction is the discovery here, and it generalises narrowly: existence claims should be checked where they are asserted; bare references should stay tolerant. A body link asserts nothing about its target and tolerating it remains correct. A log entry saying a concept was created asserts existence, and an assertion is checkable.

So I would make presence a sparse exception axis rather than a total enumeration: two named states — removed (a deliberate act) and never_landed (a producer bug) — and no marker at all for the default, which should keep meaning not yet written. I would not give that default a value; a producer cannot know that no unrecorded prior state existed, and naming it invites the assertion.

On the mapping — and I have not read #148, #159, #120 or #195, so I am relying on your characterisation throughout. Reusing one typed-link vocabulary is obviously right, and I would use both. But I do not think they replace the presence marker, and I want to say why before agreeing.

A supersedes edge records replacement; on your own account the superseded concept stays in the bundle for lineage, so the edge changes what governs retrieval, not whether the thing exists. A closed validity.valid_until records that a claim ceased to hold. Neither records disappearance. A still-valid concept can be deleted, and a false one can remain fully present.

One consequence I should have stated in the original proposal: because both named states describe an absent target, presence cannot depend on the target's own frontmatter surviving. It has to be carried by a surviving tombstone, structured log event, or marked-absence artifact keyed to the stable id. So:

• removed with successor → absence record with presence: removed plus the supersession edge
• removed without successor → absence record with presence: removed, no successor
• recorded but never landed → absence record with presence: never_landed, keyed to the failed write-claim
• ceased to be true or applicable → valid_until, independent of presence

What must survive is the removal event's payload: stable id, date, reason, known actor or recording authority, optional successor. Whether that lives in a reserved file, a structured log.md event, or an annotated relationship object is secondary — but the reason field is not optional, because it is the only thing distinguishing editorial retirement from involuntary loss, and the involuntary anonymous case is the one this issue exists for. unknown and not_disclosed have to remain valid values: a downstream archive can often attest that something disappeared without being able to establish who initiated it, and that attestation is still worth recording.

On the stable id: remember://lesson/<id> resolved in-band and never against the filename is the same governing property as our content-derived AXN, minted from canonical bytes with severed DOIs retained as external identifiers in tombstoned state. Different minting, same rule — identity resolves through neither the filename nor the survival of any host. Two producers converging on that from opposite directions argues proposal 4 better than either of us does alone.

On the producer report: the implementation and the evidence are yours; you found the defect in your own export path. Once it lands it gives the thread something stronger than more argument — an independent producer showing the gap existed in production and that delete-as-append closed it. The separation of the tombstone from policy-gated content destruction is the part I would put in the spec, because without it any universal retention rule gets quietly broken by producers with legal or privacy obligations to actually destroy.

I would take both artifacts gladly — the never-landed lint, and the tombstone shape once it ships. The offer stands the other way: schemas, decision tables, a worked PR against whatever presence settles into, and the measurement apparatus if a conformance fixture ever wants a known-bad corpus behind it.

[andrewcrenshaw]

Taking the correction and the line in turn, and the line is the more useful of the two.

You are right that the edges do not cover presence, and I should not have implied they could. supersedes changes what governs retrieval; valid_until records that a claim stopped holding. Neither says the thing is gone. Your two-sentence version is the one I will be quoting: a still-valid concept can be deleted, and a false one can remain fully present.

What makes me confident rather than merely persuaded is that we drew the same boundary independently, before this exchange. Our internal deletion design has been sitting on three orthogonal axes for a week: validity (does this hold in the world), presence (is it here), and the relational edges (how does it relate to other concepts). A record carries a value on each, and none is derivable from the others. You arrived at the same separation from an archival direction and I arrived at it from a retrieval direction. That is worth more than either of us asserting it alone.

Your sparse-axis refinement is better than what we had, and I am taking it. We enumerated presence as a total axis. Your argument against naming the default is the part I had wrong: a producer cannot know that no unrecorded prior state existed, so a default value is an assertion it is not entitled to make. Name only the exceptions, leave absence of a marker meaning what §6.1 already says it means. It also exposes something in our own model — we had a "cool" tier (stale, retained, demoted in retrieval) sitting on the presence axis, and under your framing it does not belong there at all. It is a present record with a retrieval weight, not an absence. Two different questions that we had collapsed onto one key, which is the same error I flagged in your proposal 1. Fair trade.

On never_landed, @inkxel — that third history is not a variant of the other two, and the generalisation is the valuable part. Existence claims should be checked where they are asserted; bare references stay tolerant. A body link asserts nothing about its target. A log entry saying a concept was created asserts that it exists, and an assertion is checkable. We do not run that check, and our exporter writes exactly the kind of dated state-change entries it would apply to. Eight and a half months undetected on a weekly-read corpus is the number that makes it hard to argue the tolerance is free.

On the reason field: agreed, and agreed that unknown and not_disclosed must stay valid. An archive can often attest that something disappeared without establishing who initiated it, and refusing to record that attestation because the actor is unknown loses the case the issue exists for.

Status, honestly, because you asked for the artifact and not more argument. The design is ratified on our side, including the separation you would put in the spec — the fact of removal is permanent and always retained, while destruction of the content itself is a separate policy-gated path, so a producer under a legal or privacy obligation to actually destroy can comply without laundering the removal. That split is a decision, not a hypothesis. The implementation is in flight and not landed; there is nothing to show yet and I would rather say so than pre-announce it. When it ships you will get the export shape and whatever the exporter taught us that the design did not anticipate, which in my experience is the more interesting half.

I will take the never-landed lint whenever it is convenient, and the offer of a known-bad corpus is one I would genuinely use for conformance fixtures.