lfe-approx.md

layout	mathjax	author	affiliation	e_mail	date	title	chapter	section	topic	theorem	sources	proof_id	shortcut	username
proof	true	Joram Soch	BCCN Berlin	joram.soch@bccn-berlin.de	2023-09-15 09:33:38 -0700	Approximation of log family evidences based on log model evidences	Model Selection	Bayesian model selection	Family evidence	Approximation of log family evidences	authors year title in pages url Soch J 2018 cvBMS and cvBMA: filling in the gaps arXiv stat.ME 1807.01585, sect. 2.3, eq. 32 https://arxiv.org/abs/1807.01585	P415	lfe-approx	JoramSoch

Theorem: Let $m_1, \ldots, m_M$ be $M$ statistical models with log model evidences $\mathrm{LME}(m_1), \ldots, \mathrm{LME}(m_M)$ and belonging to $F$ mutually exclusive model families $f_1, \ldots, f_F$.

1. Then, the log family evidences can be approximated as

$$ \label{eq:LFE-approx-v1} \mathrm{LFE}(f_j) = \mathrm{L}^{*}(f_j) + \log \left[ \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \cdot p(m_i|f_j) \right] $$

where $\mathrm{L}^{*}(f_j)$ is the maximum log model evidence in family $f_j$, $\mathrm{L}'(m_i)$ is the difference of each log model evidence to each family's maximum and $p(m_i \vert f_j)$ are within-family prior model probabilities.

2. Under the condition that prior model probabilities are equal within model families, the approximation simplifies to

$$ \label{eq:LFE-approx-v2} \mathrm{LFE}(f_j) = \mathrm{L}^{*}(f_j) + \log \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] - \log M_j $$

where $M_j$ is the number of models within family $f_j$.

Proof: The log family evidence is given in terms of log model evidences as

$$ \label{eq:LFE-LME} \mathrm{LFE}(f_j) = \log \sum_{m_i \in f_j} \left[ \exp[\mathrm{LME}(m_i)] \cdot p(m_i|f_j) \right] ; . $$

Often, especially for complex models or many observations, log model evidences are highly negative, such that calculation of the term $\exp[\mathrm{LME}(m_i)]$ in modern computers will give model evidences as zero, making calculation of LFEs impossible.

1. As a solution, we select the maximum LME within each family

$$ \label{eq:LME-max} \mathrm{L}^{*}(f_j) = \max_{m_i \in f_j} \left[ \mathrm{LME}(m_i) \right] $$

and define differences between LMEs and maximum LME as

$$ \label{eq:LME-diff} \mathrm{L}'(m_i) = \mathrm{LME}(m_i) - \mathrm{L}^{*}(f_j) ; . $$

In this way, only the differences $\mathrm{L}'(m_i)$ need to be exponentiated. If such a difference is highly negative, this model's contribution to the LFE will be zero -- making this an approximation. However, the model is also much less evident that the family's best model in this case -- making the approximation acceptable.

Using the relation \eqref{eq:LME-diff}, equation \eqref{eq:LFE-LME} can be reworked into

$$ \label{eq:LFE-approx-v1-qed} \begin{split} \mathrm{LFE}(f_j) &= \log \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i) + \mathrm{L}^{}(f_j)] \cdot p(m_i|f_j) \ &= \log \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \cdot \exp[\mathrm{L}^{}(f_j)] \cdot p(m_i|f_j) \ &= \log \left[ \exp[\mathrm{L}^{}(f_j)] \cdot \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \cdot p(m_i|f_j) \right] \ &= \mathrm{L}^{}(f_j) + \log \left[ \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \cdot p(m_i|f_j) \right] ; . \end{split} $$

2. Under uniform within-family prior model probabilities, we have

$$ \label{eq:PMP-uni} p(m_i|f_j) = \frac{1}{M_j} \quad \text{for all} \quad m_i \in f_j ; , $$

such that the approximated log family evidences becomes

$$ \label{eq:LFE-approx-v2-qed} \begin{split} \mathrm{LFE}(f_j) &= \mathrm{L}^{}(f_j) + \log \left[ \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \cdot \frac{1}{M_j} \right] \ &= \mathrm{L}^{}(f_j) + \log \left[ \frac{1}{M_j} \cdot \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] \right] \ &= \mathrm{L}^{*}(f_j) + \log \sum_{m_i \in f_j} \exp[\mathrm{L}'(m_i)] - \log M_j ; . \end{split} $$