Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ISDA: Irreducible Semantic Density Analysis

A framework, and a Claude Code skill, for asking a precise question of any written text:

How much of this text is doing irreducible work toward its purpose, and across which strata does that work live?

ISDA decomposes a text into seven strata organized via the Nicomachan-Boethian 2x2 classification of the classical liberal arts. Three trivium strata (Grammatica, Dialectica, Rhetorica) for form, four quadrivium strata (Arithmetica, Geometria, Musica, Astronomia) for substance. Each stratum is measured as a conditional Kolmogorov complexity against progressively richer context. Each carries a dominant intensity tag drawn from a unified four-level ladder (Stock, Implied, Selected, Original). The framework is relative to an explicit telos (what the text is trying to achieve) and a named retrieval substrate (the knowledge base against which "retrievable" is defined). It yields practical metrics for editing, originality analysis, compression, and quality assessment. Metrics that change when the telos or substrate changes, as they should.

The framework is grounded in Kolmogorov complexity (Kolmogorov 1965), Uniform Information Density (Aylett and Turk 2004; Wilmot and Keller 2020), propositional idea density (Kintsch 1974; CPIDR, Brown et al. 2008), and the recent result that language models are compressors (Delétang et al. 2023). The Claude Code skill makes it usable on any text with a single command. The Python preprocessor provides hard quantitative anchors so the analysis is not pure judgment.

What's in this repo

File Purpose Who should read it
SKILL.md The Claude Code skill definition. Analysis protocol, seven strata, intensity ladder, metrics, output template. Anyone running ISDA.
reference.md Formal grounding. Conditional-complexity definitions, the Nicomachan-Boethian 2x2 with primary-source attribution, per-stratum formal definitions, UID-at-dual-scales unification, honest caveats. Anyone who wants the math.
theory.md Longer-form theoretical paper. Kolmogorov gap, telos turn, Nicomachan-Boethian structure, seven strata as conditional complexity, unified UID spine, intensity ladder, prior art survey, LLM-as-decompressor hypothesis, open questions, references. Anyone who wants the intellectual frame.
examples.md Three worked examples. Hopkins "Pied Beauty" (Musica-Original lyric), "The Problem of Defaults" (Dialectica-Original argumentative essay), "How Photosynthesis Works" (all-Stock textbook passage), analyzed end-to-end. Anyone who wants to see ISDA in action before running it.
isda_preprocess.py Python preprocessor. Measurement-only, produces raw features for all seven strata: content-word heuristic for Arithmetica, sentence variance and syllable estimate and rhetorical-figure candidates for Musica, per-segment content-density profile for Astronomia. Anyone running ISDA or building on it.
INSTALL.md Installation for Claude Code, Claude Desktop, and standalone use. Anyone installing.
LICENSE MIT.

The core idea in three paragraphs

Kolmogorov complexity asks the wrong question. K(x) is the length of the shortest program producing string x. It is machine-independent, elegant, and uncomputable. It is also agnostic to purpose. A random string has maximum K and zero value; a cliché has low K and also zero value. Useful text occupies a middle region Kolmogorov cannot locate, because "useful" is not a property of the string.

Telos-relative complexity is the right question. Define K_τ(d, t) = min{K(x) : achieves(x, t)}: the length of the shortest document that achieves the same task as d. This binds meaning to purpose and makes the quantity tractable via task performance, embedding similarity, or human judgment. Different telos yield different answers for the same text, which is a feature. A 5,000-word essay may be bloated for "summarize the thesis" and lean for "teach this to a novice." The text is unchanged; the question is different.

Novelty is per-stratum, not global. A text's work lives across seven strata (three trivium for form, four quadrivium for substance), and at each stratum its content sits at one of four intensities: Stock (retrievable from substrate), Implied (derivable via inference rules), Selected (curatorial choice from a visible space), or Original (irreducible against full context). A lyric poem can be Musica-Original while remaining Dialectica-Stock. An argumentative essay can be Dialectica-Original while remaining Musica-Stock. A textbook passage can read Stock across all seven strata. All three are valid and informative readings.

The seven strata via the Nicomachan-Boethian 2x2

The quadrivium strata recover a classical 2x2 classification that Nicomachus of Gerasa makes explicit in Introduction to Arithmetic I.2-3 (c. 100 CE) and Boethius transmits verbatim in De Institutione Arithmetica I.1 (c. 500 CE): discrete or continuous quantity, crossed with at-rest or in-motion state.

Trivium: form (arts of the word)

# Stratum Measure
1 Grammatica `K(sentence-forms(d)
2 Dialectica `K(claims(d)
3 Rhetorica `H(decisions(d)

Quadrivium: substance (arts of number), via the 2x2

at rest in motion
discrete Arithmetica (propositional density via Kintsch and CPIDR) Musica (local prosody via Aylett-Turk UID)
continuous Geometria (document architecture) Astronomia (global trajectory via Wilmot-Keller UID)

Each stratum is a conditional Kolmogorov complexity against progressively richer context. The seven-stratum decomposition is approximate (cross-stratum interactions exist), useful enough to structure both analysis and transformation. See theory.md Part III and reference.md section 3 for full historical and formal treatment.

The intensity ladder

Each stratum carries a dominant intensity tag from a four-level gradient:

Level Name Meaning
1 Stock already in the named substrate
2 Implied in the deductive closure of substrate and inference rules
3 Selected chosen from a visible option space
4 Original irreducible against full context

The intensity is reported per-stratum as a single dominant tag, not as a full 7x4 cross-tabulation. Seven rows, two columns (conditional complexity plus intensity tag). The 28-cell matrix alternative is rejected because it compounds reliability problems, collapses empirically, and exceeds working-memory limits.

The metrics

K_τ(d, t)  ≈  Σᵢ wᵢ(t) · Sᵢ(d)                       telos-weighted complexity
SCR(d, t)  =  |d| / K_τ(d, t)                        how much length exceeds necessity
ND(d, C)   =  Σᵢ intensity_weight(Sᵢ) · |Sᵢ| / Σᵢ |Sᵢ|    weighted by stratum novelty
RI(d, KB)  =  Σᵢ retrievable_weight(Sᵢ) · |Sᵢ| / Σᵢ |Sᵢ|  weighted by stratum retrievability

Plus per-stratum dominant intensity tags, local density rollups, and a telos-aware compression curve (what fits at target lengths of 280, 800, 2,000, 5,000 characters under the stated telos).

The unified UID spine: Musica and Astronomia at dual scales

Musica and Astronomia are both measured as deviation from Uniform Information Density against a genre-conditional null, at different scales.

  • Musica = K(prosody(d) | Π_local), local-scale UID via Aylett and Turk 2004, with Cicero and Quintilian cursus tradition for humanistic provenance and Augustine De Musica I.iv.5 for the label warrant.
  • Astronomia = K(trajectory(d) | Τ_global), global-scale UID via Wilmot and Keller 2020 and Schulz et al. 2024.

The quadrivium's classical distinction within the "in motion" half (musica as number in proportion at local ratio scale, astronomia as magnitude in motion at cosmic scale) maps cleanly onto the modern UID literature's scale distinction. A single theoretical backbone, two scale instantiations. See theory.md Part V.

Worked examples

Three worked examples in examples.md, chosen to exercise the framework across its range.

Hopkins, "Pied Beauty" (80 words, curtal sonnet, public domain 1918). Proof of necessity for the Musica stratum. Musica reads Original on four of eleven lines; all four intensity levels get exercised in a single short poem. SCR ≈ 1.3 (near-minimal), with novelty concentrated in one stratum.

"The Problem of Defaults" (1,020 words, argumentative essay). Dialectica-Original in six of eleven paragraphs; Musica-Selected (rhetorical figures at ¶1, 4, 11); Astronomia-Selected (conventional argumentative trajectory). SCR ≈ 3.6. Originality is localized to inferential claims, not rhythm or arc.

"How Photosynthesis Works" (950 words, expository passage). All seven strata read Stock. SCR ≈ 28, ND = 0. The framework correctly reports that a textbook passage doing its job has zero originality at every stratum. SCR > 20 is the right answer for pedagogical elaboration, not a bloat signal.

Installation

See INSTALL.md for Claude Code, Claude Desktop, and standalone Python usage.

For Claude Code:

git clone https://github.com/vincitamore/ISDA.git ~/.claude/skills/isda

In any Claude Code session, invoke the skill: "run ISDA on this text, telos is X, substrate is Y".

Philosophy

The framework rests on three operational claims.

First, there is no such thing as text density in the abstract. There is only density relative to a stated purpose and a named substrate. A framework that reports "how dense is this text?" without naming a telos and substrate is hiding its assumptions.

Second, novelty is not a stratum; it is a per-stratum property. A text can be original in its rhythm, its inference, its arrangement, its trajectory, its architecture, or its atomic content, each independently. The intensity ladder exposes where the novelty lives.

Third, measurement is not judgment. The preprocessor provides hard counts; the stratum classification and intensity tagging are semantic work that requires understanding. The framework makes the boundary between the two explicit.

The numbers are estimates (K and K_τ are uncomputable), but the estimates are principled and the inputs are visible.

When to use ISDA

Editing your own writing. Find which stratum carries your novelty and which strata are scaffolding; cut at the right level.

Originality analysis. Detect at which stratum a text adds something new against a named corpus, and confirm that other strata are correctly at Stock.

Compression targets. Produce tweet, paragraph, abstract, or essay versions that preserve the telos under the stratum profile.

Quality assessment. Is the intensity profile appropriate for the genre? Is the SCR in the genre-qualified band?

Editorial feedback. "Your Dialectica is Original in ¶1, 3, 5, 7, 9; your Musica is Selected; your Astronomia is Stock; the essay's work lives in the inferential moves, not the rhythm or the arc" is actionable in a way "the paper is too long" is not.

Prompt compression. Treat the prompt as d, the desired completion as t, minimize K_τ(prompt, t) empirically with per-stratum weights reflecting which strata the completion actually needs.

Curriculum design. Measure how K_τ and per-stratum intensities change as the assumed reader's substrate grows.

When not to use ISDA

As a judgment on rhetorical or aesthetic value. The framework measures semantic load against a stated task. A perfectly efficient text can still be unpersuasive; an inefficient text can still be artful. Use alongside other criteria, not instead of them.

Without naming a telos and a substrate. The numbers will look rigorous and mean nothing.

As a rigid editor. The analysis informs judgment; it does not replace it. Where stratum boundaries fuzz (e.g., Grammatica and Musica on metrical verse), be explicit about where you draw the line.

References

  • Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information.
  • Shannon, C. E. (1948). A mathematical theory of communication.
  • Nicomachus of Gerasa (c. 100 CE). Introduction to Arithmetic, I.2-3 (D'Ooge trans. 1926).
  • Boethius (c. 500 CE). De Institutione Arithmetica, I.1 (Masi trans. 1983).
  • Aylett, M. and Turk, A. (2004). The smooth signal redundancy hypothesis. Language and Speech 47(1).
  • Levy, R. and Jaeger, T. F. (2007). Speakers optimize information density through syntactic reduction. NIPS.
  • Kintsch, W. (1974). The Representation of Meaning in Memory.
  • Brown, C. et al. (2008). Automatic measurement of propositional idea density from part-of-speech tagging. (CPIDR)
  • Wilmot, D. and Keller, F. (2020). Modelling suspense in short stories as uncertainty reduction over neural representation. ACL.
  • Schulz, P., Patrício, C. and Odijk, J. (2024). Narrative Information Theory. NeurIPS Workshop, arXiv:2411.12907.
  • Reagan, A. J. et al. (2016). The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science.
  • Delétang, G. et al. (2023). Language modeling is compression. arXiv:2309.10668.
  • Cilibrasi, R. and Vitányi, P. (2005). Clustering by compression.
  • Barthes, R. (1970). S/Z.
  • Coh-Metrix (Graesser, McNamara et al. 2004, 2011); Biber, D. (1988) Variation Across Speech and Writing.

Full reference list in theory.md.

License

MIT. See LICENSE. Copy it, fork it, ship it, improve it. Attribution is appreciated but not required.

Status

Exploratory first release. The formal definitions are proposals, not established theorems. Issues, counter-examples, and improvements welcome, especially worked examples on texts from domains outside the ones the current examples cover (scientific writing, legal reasoning, code, translation, multilingual work). CPIDR's English-specific POS rules mean non-English Arithmetica support is a known limitation.

About

How much of this text is doing irreducible work toward its purpose, and across which strata does that work live?

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages