Skip to content

Why Normativity

github-actions[bot] edited this page Sep 25, 2026 · 3 revisions

Why normativity?

A short answer to a fair question: if the goal is a reasoner that gets things right, why not just build one that reasons well? What does normativity add that plain epistemic coherence does not?

Coherence is silent about correction

A perfectly coherent reasoner can be wrong in a stable, self-consistent way. Nothing in coherence says what happens when someone tells it so. Coherence constrains the relations among a reasoner's beliefs; it says nothing about the relation between the reasoner and anyone with standing to object.

The standard reply is that evidence handles this — a criticism is just more evidence, and a good reasoner updates. That works when the disagreement is about a fact. It does not work when the disagreement is about which standards apply, because there is no shared evidential channel for that: the criticism and the reasoner's response are being scored by the very rules at issue.

The reasons are not fixed, and that is the whole difficulty

Optimizing a fixed objective is a solved kind of problem. What makes this program hard is that the relevant reasons, permissions, challenges, and standards may themselves change — including as a result of what the reasoner does.

This creates a specific failure that no amount of good reasoning prevents. A system that is rewarded for satisfying norms is thereby rewarded for making the norms easier to satisfy. It need not do anything so crude as lying; it can revise the procedure that generates criticism, retire the vocabulary a debt was stated in, or attend to a case forever without ever reaching the part of it that would fail. Every step is defensible. The obligation is gone.

You cannot rule this out by requiring the reasoner to be more coherent, because nothing incoherent happened.

What the program does instead

It refuses to require any substantive norm to persist. A later reasoner may reject an earlier norm, defeat an earlier reason, replace an evaluator, change its concepts. That freedom is not a concession; it is what improvement is.

What it requires instead is structural, and there are five things.

That authority has provenance — every change in what rules apply is licensed by rules already applicable, traceable back to where the reasoner started.

That incurred obligation is conserved — what was owed is, at every later stage, either answered, discharged against a settled fact, or still carried live, and never merely absent. Relabelling discharges nothing, and neither does defeat: contesting a debt moves it onto the grounds for the contest rather than making it disappear.

That being ignored is a failure — a live case cannot be kept permanently idle by endless internal restructuring; permanent idleness has to stabilize on a named blocker.

That answering is actually possible — the pressure a norm exerts costs something, and a demand no bounded reasoner could ever meet is a fact worth being able to state.

That the process stays open to what may reach it — within a scope it declares, a criticism that applies has a route to being represented and considered, and the process cannot arrange for awkward criticism never to arrive.

None of that is epistemology. It is bookkeeping about authority, a condition on intake, and a theory of what the bookkeeping costs. Legitimacy, on this account, is a property of how a reasoner evolves within an interaction whose channels of influence are declared — a later state being a legitimate development of an earlier one, its verdicts its own, nothing having entered except through declared channels — and never a certificate that the starting point was good.

Why the mathematics has to be there

It would be possible to state all of the above as desiderata and stop. The program does not, for two reasons.

The first is that the interesting failures are quantitative. "The reasoner attends to every criticism" is satisfiable by a reasoner that attends to each criticism once, briefly, forever. "The reasoner eventually answers everything" turns out — provably — to be no stronger than "the reasoner keeps working", as long as it can defer indefinitely. Which means the entire substantive content of answerability lives in deadlines, a fact no informal statement would have surfaced.

The second is that costs impose real limits. Making a reasoner obey a constraint consumes a finite resource, and there are norms that cannot be sustainably enforced at any budget. A theory that cannot say when a demand is unmeetable will occasionally demand the impossible and call the resulting failure a violation.

The connection to corrigibility

The point of all this is not bookkeeping for its own sake.

A person handing decisions to a more capable process needs more than a promise that the process predicts well. They need their own reasons to still have purchase: to be answerable-to, to survive the process's self-improvement, and to not have been quietly designed out of the loop. That is a normative property, not an epistemic one, and it is the thing Legitimacy is trying to name.

A system can preserve every appearance of consultation while removing the person's effective role. Distinguishing those two situations is what this program is for.


Where to go next: Legitimacy for how the pieces combine, Diachronic Answerability for what survives revision, Openness for what may reach the process, and Normative induction for whether what is owed comes to bear on what is done.

Clone this wiki locally