TSP — Information Thermodynamics of Intelligence

A unified framework connecting energy, structure, and capability emergence in neural networks.

This repository accompanies the working paper Information Thermodynamics of Intelligence: A Unified Framework Connecting Energy, Structure, and Capability Emergence (v1.4) by Antreas Antoniou (Axiotic AI).

The PDF is at info-thermo-intelligence-v1.4.pdf.

The argument

Intelligence requires computation; computation requires energy. The brain runs on ~20 W and exhibits compositional reasoning, long-horizon planning, and real-time sensorimotor control. Frontier language model training and inference run at ~10⁶× the energy per comparable cognitive operation. Parameter and data scaling do not close this gap. Something about the structure of computation does.

Four research programs have been independently converging on the same insight from different directions:

• Prediction–dissipation (Still et al., 2012): thermodynamic dissipation equals retained non-predictive information.
• Information bottleneck (Tishby et al., 1999): learning is compression of input that preserves predictive information about the target.
• Free energy principle (Friston, 2010): biological intelligence minimises variational free energy.
• Compression thesis (Delétang et al., 2023): language modelling is compression; capable models are universal lossless compressors.

Each holds the architecture of the computational system fixed.

The Thermodynamic Structure Principle

We promote architecture to a first-class variable. The TSP states: intelligence is the process of converting free energy into predictive compression of environmental structure, and architectural topology determines the efficiency of this conversion.

Formally, we define a generalised free energy functional over parameters θ and architecture 𝒜:

ℱ[θ, 𝒜] = E_comp(θ, 𝒜) + T_eff · D_KL(q_θ ‖ p₀) − T_eff · I(Y; T_θ)

with architecture decomposed along three axes:

1. Connectivity topology 𝒢 — graph structure of information flow
2. Memory architecture ℳ — how information persists across time/layers
3. Compression geometry 𝒞 — information-geometric structure of representation space

Minimising ℱ recovers each of the four programs as a special case:

Program	Recovery
Information bottleneck (Tishby)	Fix 𝒜, optimise compression–prediction trade-off
Free energy principle (Friston)	Take T_eff → 0
Prediction–dissipation (Still et al.)	Decompose I(X; T_θ) into predictive and non-predictive components
Compression / MDL (Rissanen; Delétang et al.)	Marginalise over θ at fixed 𝒜, T_eff = 1

Architecture as channel capacity

We formalise architectural channel capacity as the supremum of predictive information achievable under a given parameter and energy budget:

C_𝒜(N, E) = sup { I(Y; T_θ) | E_comp(θ, 𝒜) ≤ E }

This yields three regimes:

• Subcritical (C_𝒜 < I_min): insufficient capacity, capability at chance.
• Critical (C_𝒜 ≈ I_min): capacity meets requirement; capability undergoes a discontinuous jump; energy-to-competence is minimised.
• Supercritical (C_𝒜 ≫ I_min): capacity vastly exceeds requirement; uniformly high capability; higher energy than the optimum.

Empirical validation

The framework is grounded in over 360 controlled experiments across seven hypothesis groups (H1–H7), spanning synthetic algorithmic tasks, real-world NLP benchmarks, and direct GPU energy measurement on RTX 4090 via NVML.

Headline findings:

ID	Claim	Evidence
H1	Sharp phase transitions in capability at critical topology thresholds p*	Copy/sort p* ≈ 0.75–0.80; reverse p* = 1.0; 0% → 100% jumps
H2	Learned topology achieves 36% lower mean connectivity at matched capability	Hybrid PPA, per-head learnable p
H3	Memory and topology are thermodynamically substitutable resources	Titans surprise-driven memory solves reverse at p = 0.5 (impossible without memory)
H4	No topology dependence in supercritical regime	NER 86.4–87.3% F1 and SST-2 91.2–91.8% across p ∈ [0.40, 2.00]
H4E	Critical topology minimises energy-to-competence	Among reliable topologies, p* saves 7–26% energy; 3,077 J at p* = 0.80

Experimental stack: Power-Based Partial Attention (PPA) Transformer parameterising attention as O(L^{1+p}), p ∈ [0, 2]. Two model scales (604K and 11M params). Synthetic tasks (copy/reverse/sort/associative recall/induction) plus WikiANN NER and SST-2 sentiment. Energy via NVIDIA Management Library.

Five novel predictions

The TSP yields five experimentally falsifiable predictions:

1. P1 — Energy scaling with topology. Total training energy is U-shaped with a minimum at p*; diverges below p*.
2. P2 — Learned topology minimises dissipation. Converged learned per-head topology minimises non-predictive information and thus dissipation (W_diss = k_B T · I_nonpred).
3. P3 — Task information from critical topology. I_min ≥ Φ(p*), giving a thermodynamic lower bound on task information content from observed phase transitions.
4. P4 — Structural scaling laws. Capability follows a structural scaling distinct from parameter-only, with p*(N) decreasing as N grows.
5. P5 — Memory–topology substitution surface. Memory M and connectivity p lie on iso-capability surfaces with negative slope; ∂p/∂M|_cap < 0.

Contributions

1. A variational framework unifying four major research programs as special cases of a generalised free energy functional over both parameters and architecture.
2. Formalisation of architecture as a channel capacity that controls energy-to-intelligence conversion efficiency.
3. Derivation of conditions under which capability emergence corresponds to a thermodynamic phase transition.
4. Over 360 controlled experiments providing the first systematic empirical evidence connecting architectural topology to thermodynamic phase transitions in capability.
5. Five novel testable predictions about energy scaling, dissipation minimisation, and structural scaling laws.

Connection to neuroscience

The TSP predicts that biological circuits should exhibit minimalist connectivity operating near critical thresholds, surprise-driven memory consolidation as thermodynamically optimal compression, and hierarchical coarse-graining as iterated channel capacity reduction. These align with known properties of sparse cortical connectivity, hippocampal novelty signalling, and hierarchical compression in visual cortex.

Citation

@misc{antoniou2026tsp,
  title  = {Information Thermodynamics of Intelligence: A Unified Framework
            Connecting Energy, Structure, and Capability Emergence},
  author = {Antoniou, Antreas},
  year   = {2026},
  note   = {Working paper v1.4, Axiotic AI}
}

Contact

Antreas Antoniou — antreas@axiotic.ai — Axiotic AI

---

Intelligence is not the size of the fire — it is the shape of the forge.