Entropy

[dfoxfranke]

This patch set specifies the typeclasses 'Entropy' and 'MaybeEntropy', and defines instances for every distribution.

[Shimuuar]

Thank you! I have couple of remarks though

1. Is using one type class for Shannon entropy and differential entropy a good idea? Most importantly could both be used in some context?
2. Instance for LinearTransfrom is buggy. Differential entropy isn't invariant under linear transformation and Shannon is. I think it's argument for splitting type classes.

[dfoxfranke]

1. I believe that it's appropriate to combine them: for discrete distributions, the two always coincide, and for continuous distributions, Shannon entropy is ill-defined. I admit, though, that I've never had much practical use for differential entropy and I find some of its properties highly unintuitive, so perhaps I'm missing something.
2. Look again. That's the reason I put (DiscreteDistr d) in the instance context.

[dfoxfranke]

Actually, let me walk back that "always coincide" claim. I've always thought of differential entropy as a generalization of Shannon entropy, but that's not really true, or at least would require something beyond standard real analysis to justify. Instead, I should substitute the weaker claim that there is no situation in which differential entropy and Shannon entropy are both defined and unequal.

[Shimuuar]

1. That sounds reasonable
2. Sorry missed that.

Merged. Thank you!