prior-maxent.md

layout	mathjax	author	affiliation	e_mail	date	title	chapter	section	topic	definition	sources	def_id	shortcut	username
definition	true	Joram Soch	BCCN Berlin	joram.soch@bccn-berlin.de	2020-12-02 10:13:00 -0800	Maximum entropy prior distribution	General Theorems	Bayesian statistics	Prior distributions	Maximum entropy priors	authors year title in pages url Wikipedia 2020 Prior probability Wikipedia, the free encyclopedia retrieved on 2020-12-02 https://en.wikipedia.org/wiki/Prior_probability#Uninformative_priors	D121	prior-maxent	JoramSoch

Definition: Let $m$ be a generative model with likelihood function $p(y \vert \theta, m)$ and prior distribution $p(\theta \vert \lambda, m)$ using prior hyperparameters $\lambda$. Then, the prior distribution is called a "maximum entropy prior", if

1. when $\theta$ is a discrete random variable, it maximizes the entropy of the prior probability mass function:

$$ \label{eq:prior-maxent-disc} \lambda_{\mathrm{maxent}} = \operatorname*{arg,max}_{\lambda} \mathrm{H}\left[ p(\theta \vert \lambda, m) \right] ; ; $$

2. when $\theta$ is a continuous random variable, it maximizes the differential entropy of the prior probability density function:

$$ \label{eq:prior-maxent-cont} \lambda_{\mathrm{maxent}} = \operatorname*{arg,max}_{\lambda} \mathrm{h}\left[ p(\theta \vert \lambda, m) \right] ; . $$