Mathematical Framework

Mathematical Framework

This page details the mathematical foundations of entropic measurement, including the concepts of entropy, divergence, and uncertainty quantification. We provide clear definitions, key equations, and pseudocode for computation.

1. Entropy

Definition:
Entropy is a quantitative measure of uncertainty in a probability distribution P. For a discrete random variable X with outcomes x₁, x₂, ..., xₙ and probability mass function P(x):

H(P) = -Σ₁ⁿ P(xᵢ) log P(xᵢ)

1.2 Example Pseudocode

def entropy(probabilities): import math return -sum(p * math.log(p) for p in probabilities if p > 0)

2. Divergence

Kullback-Leibler (KL) Divergence:
KL divergence quantifies how one probability distribution Q diverges from a reference distribution P:

D_KL(P || Q) = Σ₁ⁿ P(xᵢ) log[P(xᵢ)/Q(xᵢ)]

Example Pseudocode

def kl_divergence(P, Q):
    import math
    return sum(p * math.log(p/q) for p, q in zip(P, Q) if p > 0 and q > 0)

3. Uncertainty Quantification

• Shannon Entropy:
  H(P) = -Σ P(x) log P(x)
• KL Divergence:
  D_KL(P || Q) = Σ P(x) log [P(x) / Q(x)]
• Jensen-Shannon Divergence:
  JS(P, Q) = 1/2 D_KL(P || M) + 1/2 D_KL(Q || M)
  where M = 1/2 (P + Q)

4. References

• Shannon, C. E. (1948). A Mathematical Theory of Communication.
• Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory.