Glossary

Glossary

A quick-reference guide to the key terms and definitions used throughout the entropic_measurement project and its documentation.

Entropy
The quantitative measure of uncertainty, disorder, or information in a random variable or dataset.

Shannon Entropy (H)
The standard measure of information content introduced by Claude Shannon. Represents the average amount of "surprise" in a probability distribution.

Rényi Entropy
A generalization of Shannon entropy parameterized by order α; useful for tuning sensitivity to rare or frequent events.

Tsallis Entropy
A family of entropy measures used in non-extensive statistical mechanics, parameterized by q.

Mutual Information (MI)
Measures the amount of information shared by two variables.

Conditional Entropy
The remaining uncertainty in one variable given knowledge of the other.

KL Divergence (Kullback–Leibler Divergence)
Asymmetric measure of how one probability distribution diverges from a reference distribution.

Cross Entropy
Measures the average number of bits needed to encode data from a target distribution using a given probability model.

Bias Correction
Methods like Miller-Madow or Grassberger employed to correct the bias in entropy estimates from finite data.

Smoothing
Techniques (e.g., Laplace or Lidstone) for adjusting probability distributions to handle zero frequencies or sparse data.

Unit (of Entropy)
"Bits" if using log2, "nats" if using natural logarithm, "dits" for log10.

Data Binning
Grouping continuous or high-cardinality data into discrete categories for analysis.

Streaming Entropy
Entropy estimation over large datasets by processing data in chunks, without needing to load everything into memory.

API
Application Programming Interface—the set of public functions, classes, and parameters offered by the library.

Configuration File
YAML or JSON files used to specify settings or parameters for a run.