-
Notifications
You must be signed in to change notification settings - Fork 1
/
info.json
17 lines (17 loc) · 1.65 KB
/
info.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"abstract": "We consider the problem of estimating Shannon's entropy $H$ from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior distribution over the space of countably infinite discrete distributions, and has found major applications in Bayesian non- parametric statistics and machine learning. Here we show that it provides a natural family of priors for Bayesian entropy estimation, due to the fact that moments of the induced posterior distribution over $H$ can be computed analytically. We derive formulas for the posterior mean (Bayes' least squares estimate) and variance under Dirichlet and Pitman-Yor process priors. Moreover, we show that a fixed Dirichlet or Pitman-Yor process prior implies a narrow prior distribution over $H$, meaning the prior strongly determines the entropy estimate in the under-sampled regime. We derive a family of continuous measures for mixing Pitman-Yor processes to produce an approximately flat prior over $H$. We show that the resulting \"Pitman-Yor Mixture\" (PYM) entropy estimator is consistent for a large class of distributions. Finally, we explore the theoretical properties of the resulting estimator, and show that it performs well both in simulation and in application to real data.",
"authors": [
"Evan Archer",
"Il Memming Park",
"Jonathan W. Pillow"
],
"id": "archer14a",
"issue": 81,
"pages": [
2833,
2868
],
"title": "Bayesian Entropy Estimation for Countable Discrete Distributions",
"volume": 15,
"year": 2014
}