Roberto maestre - rmaestre@gmail.com
Bojan Mihaljevic - boki.mihaljevic@gmail.com

https://controls.engin.umich.edu/wiki/index.php/Correlation_and_Mutual_Information

Mutual information (also referred to as transinformation) is a quantitative measurement of how much one random variable (Y) tells us about another random variable (X). In this case, information is thought of as a reduction in the uncertainty of a variable. Thus, the more mutual information between X and Y, the less uncertainty there is in X knowing Y or Y knowing X. For our purposes, within any given process, several parameters must be selected in order to properly run the process. The relationship between variables is integral to correctly determine working values for the system. For example, adjusting the temperature in a reactor often causes the pressure to change as well. Mutual information is most commonly measured in logarithms of base 2 (bits) but is also found in base e (nats) and base 10 (bans).


data:
[ (0, 0, 1, 1, 0, 1, 1, 2, 2, 2),
  (3, 4, 5, 5, 3, 2, 2, 6, 6, 1),
  (7, 2, 1, 3, 2, 8, 9, 1, 2, 0),
  (7, 7, 7, 7, 7, 7, 7, 7, 7, 7),
  (0, 1, 2, 3, 4, 5, 6, 7, 1, 1) ]


./it_tool.py

Entropy(X_1): 0.759176
Elapsed time: 0.000941

Entropy(X_3): 0.000000
Elapsed time: 0.000046

Entropy(X_4): 0.856864
Elapsed time: 0.000247

Entropy(X_0, X_1): 0.759176
Elapsed time: 0.000639

Entropy(X_3, X_3): 0.000000
Elapsed time: 0.000082

MI(X_0, X_1): 0.472903
Elapsed time: 0.001174

MI(X_1, X_2): 0.555834
Elapsed time: 0.002696