Skip to content

Advanced Features

Jump to bottom
Raphael Constantinis edited this page Jul 23, 2025 · 2 revisions

Advanced Features

This page covers advanced entropy measurement techniques and features available in the entropic_measurement library.

Advanced Entropy Methods

Conditional Entropy

Conditional entropy measures the uncertainty of one variable given knowledge of another variable. It's particularly useful for understanding information dependencies.

from entropic_measurement import conditional_entropy

# Calculate H(Y|X)
entropy_y_given_x = conditional_entropy(y_data, x_data)
print(f"Conditional entropy H(Y|X): {entropy_y_given_x:.4f}")

Mutual Information

Mutual information quantifies the amount of information shared between two variables.

from entropic_measurement import mutual_information

# Calculate mutual information between X and Y
mi = mutual_information(x_data, y_data)
print(f"Mutual information I(X;Y): {mi:.4f}")

Cross Entropy

Cross entropy measures the difference between two probability distributions.

from entropic_measurement import cross_entropy

# Calculate cross entropy between predicted and actual distributions
ce = cross_entropy(predicted_dist, actual_dist)
print(f"Cross entropy: {ce:.4f}")

Relative Entropy (KL Divergence)

Kullback-Leibler divergence measures how one probability distribution differs from another.

from entropic_measurement import kl_divergence

# Calculate KL divergence D(P||Q)
kl_div = kl_divergence(p_distribution, q_distribution)
print(f"KL divergence D(P||Q): {kl_div:.4f}")

Custom Corrections

Bias Corrections

The library provides several bias correction methods for entropy estimation:

Miller-Madow Correction

from entropic_measurement import shannon_entropy

# Apply Miller-Madow bias correction
entropy_corrected = shannon_entropy(data, correction='miller_madow')

Grassberger Correction

# Apply Grassberger correction for small samples
entropy_corrected = shannon_entropy(data, correction='grassberger')

Custom Correction Functions

def custom_correction(entropy, sample_size, alphabet_size):
    """Custom bias correction function"""
    return entropy + (alphabet_size - 1) / (2 * sample_size)

# Use custom correction
entropy_corrected = shannon_entropy(data, correction=custom_correction)

Smoothing Techniques

Smoothing helps handle zero probabilities in sparse data:

# Laplace smoothing (add-one smoothing)
entropy_smooth = shannon_entropy(data, smoothing='laplace', alpha=1.0)

# Lidstone smoothing with custom alpha
entropy_smooth = shannon_entropy(data, smoothing='lidstone', alpha=0.5)

Combining Multiple Measures

Entropy Profiles

Create comprehensive entropy profiles for complex data analysis:

from entropic_measurement import EntropyProfile

# Create entropy profile with multiple measures
profile = EntropyProfile(data)
results = profile.compute_all()

print(f"Shannon entropy: {results['shannon']:.4f}")
print(f"Rényi entropy (α=2): {results['renyi_2']:.4f}")
print(f"Tsallis entropy (q=2): {results['tsallis_2']:.4f}")
print(f"Hartley entropy: {results['hartley']:.4f}")

Weighted Combinations

Combine multiple entropy measures with custom weights:

from entropic_measurement import weighted_entropy_combination

entropies = {
    'shannon': shannon_entropy(data),
    'renyi': renyi_entropy(data, alpha=2),
    'tsallis': tsallis_entropy(data, q=2)
}

weights = {'shannon': 0.5, 'renyi': 0.3, 'tsallis': 0.2}
combined_entropy = weighted_entropy_combination(entropies, weights)

Multi-dimensional Analysis

Analyze entropy across multiple dimensions or variables:

from entropic_measurement import multidimensional_entropy

# For multivariate data
entropy_2d = multidimensional_entropy(data_2d, method='joint')
entropy_marginal = multidimensional_entropy(data_2d, method='marginal')

Performance Optimization

Parallel Processing

Leverage parallel processing for large datasets:

from entropic_measurement import shannon_entropy

# Enable parallel processing
entropy_parallel = shannon_entropy(large_data, parallel=True, n_jobs=4)

Memory-Efficient Computation

For very large datasets, use streaming computation:

from entropic_measurement import StreamingEntropy

# Initialize streaming entropy calculator
streaming_calc = StreamingEntropy()

# Process data in chunks
for chunk in data_chunks:
    streaming_calc.update(chunk)

final_entropy = streaming_calc.finalize()

Advanced Configuration

Custom Base Logarithms

# Use different logarithm bases
entropy_log2 = shannon_entropy(data, base=2)  # bits
entropy_ln = shannon_entropy(data, base='e')   # nats
entropy_log10 = shannon_entropy(data, base=10) # dits

Precision Control

# High precision computation
from decimal import Decimal
entropy_precise = shannon_entropy(data, precision=Decimal)

Further Resources

Documentation

  • API Reference - Complete function and class documentation
  • Examples - Practical usage examples
  • Theory - Mathematical foundations of entropy measures

External Resources

Research Papers

  • Paninski, L. (2003). "Estimation of entropy and mutual information" - Bias correction methods
  • Grassberger, P. (1988). "Finite sample corrections to entropy and dimension estimates" - Advanced corrections
  • Schürmann, T. (2004). "Bias analysis in entropy estimation" - Comprehensive bias analysis

Community

Last updated: July 23, 2025

Clone this wiki locally