Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
300 lines (235 sloc) 13.9 KB
The \eslmod{histogram} module is for collecting scores, fitting
them to expected distributions, and displaying them.
The histogram automatically reallocates its bins as data points
arrive, so the caller only needs to provide some initial guidance
about bin size and ``phase'' (offset of the bins relative to the real
number line). It accumulates counts in 64-bit unsigned integers, so
it can handle over $10^19$ total counts. Optionally (and provided
that the caller knows it has enough memory to support this), a
``full'' histogram can be created and used to collect a sorted vector
of raw (unbinned) values.
Various different ways of fitting histogram data to different sorts of
expected distributions are supported, with interfaces to all of
Easel's statistical distribution modules. Data fitting is oriented
toward the case where the values are scores, with high scores being of
the most interest; for instance, routines for obtaining and fitting
the right (high-scoring) tail are provided, but not for the left tail.
Several of the output functions output data as XY data files suitable
for input into the popular and freely available \prog{xmgrace}
graphing program [\url{http://plasma-gate.weizmann.ac.il/Grace/}].
The API for the \eslmod{histogram} module is summarized in
Table~\ref{tbl:histogram_api}.
\begin{table}[hbp]
\begin{center}
{\small
\begin{tabular}{|ll|}\hline
\apisubhead{Collecting data in an \ccode{ESL\_HISTOGRAM}}\\
\hyperlink{func:esl_histogram_Create()}{\ccode{esl\_histogram\_Create()}} & Create a new \ccode{ESL\_HISTOGRAM}.\\
\hyperlink{func:esl_histogram_CreateFull()}{\ccode{esl\_histogram\_CreateFull()}} & A \ccode{ESL\_HISTOGRAM} to keep all data samples.\\
\hyperlink{func:esl_histogram_Destroy()}{\ccode{esl\_histogram\_Destroy()}} & Frees a \ccode{ESL\_HISTOGRAM}.\\
\hyperlink{func:esl_histogram_Add()}{\ccode{esl\_histogram\_Add()}} & Add a sample to the histogram.\\
\apisubhead{Declarations about binned data, before fitting}\\
\hyperlink{func:esl_histogram_DeclareCensoring()}{\ccode{esl\_histogram\_DeclareCensoring()}} & Collected data were left-censored.\\
\hyperlink{func:esl_histogram_DeclareRounding()}{\ccode{esl\_histogram\_DeclareRounding()}} & Declare collected data were no more accurate than bins.\\
\hyperlink{func:esl_histogram_SetTail()}{\ccode{esl\_histogram\_SetTail()}} & Declare only tail $>$ some threshold is considered "observed".\\
\hyperlink{func:esl_histogram_SetTailByMass()}{\ccode{esl\_histogram\_SetTailByMass()}} & Declare only right tail mass is considered "observed".\\
\apisubhead{Accessing raw data samples}\\
\hyperlink{func:esl_histogram_GetRank()}{\ccode{esl\_histogram\_GetRank()}} & Retrieve n'th high score.\\
\hyperlink{func:esl_histogram_GetData()}{\ccode{esl\_histogram\_GetData()}} & Retrieve vector of all raw scores.\\
\hyperlink{func:esl_histogram_GetTail()}{\ccode{esl\_histogram\_GetTail()}} & Retrieve all raw scores above some threshold.\\
\hyperlink{func:esl_histogram_GetTailByMass()}{\ccode{esl\_histogram\_GetTailByMass()}} & Retrieve all raw scores in right tail mass.\\
\apisubhead{Setting expected counts}\\
\hyperlink{func:esl_histogram_SetExpect()}{\ccode{esl\_histogram\_SetExpect()}} & Set expected counts for complete distribution.\\
\hyperlink{func:esl_histogram_SetExpectedTail()}{\ccode{esl\_histogram\_SetExpectedTail()}} & Set expected counts for right tail.\\
\apisubhead{Output}\\
\hyperlink{func:esl_histogram_Write()}{\ccode{esl\_histogram\_Write()}} & Print a "pretty" ASCII histogram.\\
\hyperlink{func:esl_histogram_Plot()}{\ccode{esl\_histogram\_Plot()}} & Output a histogram in xmgrace XY format.\\
\hyperlink{func:esl_histogram_PlotSurvival()}{\ccode{esl\_histogram\_PlotSurvival()}} & Output $P(X>x)$ in xmgrace XY format.\\
\hyperlink{func:esl_histogram_PlotQQ()}{\ccode{esl\_histogram\_PlotQQ()}} & Output a Q-Q plot in xmgrace XY format.\\
\hyperlink{func:esl_histogram_Goodness()}{\ccode{esl\_histogram\_Goodness()}} & Evaluate fit between observed, expected. \\
\hline
\end{tabular}
}
\end{center}
\caption{The \eslmod{histogram} API.}
\label{tbl:histogram_api}
\end{table}
\subsection{Example of using the histogram API}
The example code below stores 10,000 samples from a Gumbel
distribution in a histogram, retrieves a vector containing the sorted
samples, fits a Gumbel distribution to that dataset, sets the expected
counts in the histogram, prints the observed and expected counts in an
ASCII histogram, and evaluates the goodness-of-fit.
\input{cexcerpts/histogram_example}
Some points of interest:
\begin{itemize}
\item When the histogram is created, the arguments \ccode(-100, 100, 0.5)
tell it to bin data into bins of width 0.5, initially
starting at -100 and ending at 100. This initialization
is described below (see ``Specifying binning of data values'').
\item Samples are collected one at a time with
\ccode{esl\_histogram\_Add()}.
\item After the data have been collected in a \emph{full} histogram, a
vector of sorted raw data values can be retrieved using functions
like \ccode{esl\_histogram\_GetData()}, and used to fit parameters
of an expected distribution to the data.
\item In addition to the observed binned counts, you can optionally
set \emph{expected} binned counts in the histogram by calling
\ccode{esl\_histogram\_SetExpect()} and providing pointers
to an appropriate distribution function and its parameters.
\item The \ccode{esl\_histogram\_Print()} function shows an ASCII text
representation of the observed counts (and expected counts, if set)
that looks a lot like FASTA's nice histogram output.
\item The \ccode{esl\_histogram\_Goodness()} function compares the
observed and expected binned counts, and calculates two goodness of
fit tests: a G-test, and a $\chi^2$ test.
\end{itemize}
\subsection{Specifying binning of data values}
The histogram collects data values into bins. When the histogram is
created, the bin width and the relative offset of the bins is
permanently set, and an initial range is allocated.
For example, the call \ccode{esl\_histogram\_Create(-10, 10, 0.5)}
creates 40 bins of width 0.5 from -10 to 10, with the first bin
collecting scores from $-10 < x \leq -9.5$, and the last bin
collecting scores $9.5 < x \leq 10.0$.
The lower bound of the initialization permanently sets the relative
offset of the bins. That is, \ccode{esl\_histogram\_Create(-10, 10,
0.5)} makes the first bin $-10 < x \leq -9.5$, whereas
\ccode{esl\_histogram\_Create(-10.1, 9.9, 0.5)} makes the first bin
$-10.1 < x \leq -9.6$.
Aside from that, the initial range is only a suggestion. You can add
any real-valued $x$ to the histogram. The histogram will silently
reallocate itself to a wider range as needed. The ability of a
histogram to store data is effectively unlimited. Up to $2^{64}-1$
(more than $10^{19}$) counts can be collected. The histogram requires 16
bytes of storage per bin, and the number of bins it allocates scales
as $x_{\mbox{max}} - x_{\mbox{min}} / w$.
\subsection{Optional collection of raw data values: full histograms}
Normally a histogram would store only binned counts, so it can
efficiently summarize even very large numbers of samples.
In some cases it is useful to keep a list of the raw data values --
for instance, for more accurate parameter fitting to expected
distributions. This can be done by creating a ``full'' histogram with
\ccode{esl\_histogram\_CreateFull()} instead of
\ccode{esl\_histogram\_Create()}. (The example code above did this,
because it did parameter fitting to the raw data.) After data have
been collected in a full histogram, individual raw values or pointers
to sorted arrays of raw values can be retrieved using the
\ccode{esl\_histogram\_Get*} functions.
A full histogram may require much more memory: about 4 bytes per data
point. You may not want to use full histograms if your problem
involves collecting many ($> 10^9$, say) data points.
\subsection{Different parameter fitting scenarios}
By default, the data you collect are assumed to be \emph{complete}.
You observed all samples; if you fit to any expected distribution, the
expected distribution is assumed to describe the complete data; the
parameters of the expected distribution are to be fitted to an array
of the complete raw data samples; and any goodness of fit test is to
be applied to the complete data. This is the simplest, most obvious
case.
Other situations may arise. In addition to complete data, Easel is
designed to deal with four other cases:
\begin{enumerate}
\item The collected data are complete, and they are fit to a
distribution that describes the complete data, but parameter
fitting is done only in the right (highest-scoring) tail. This
makes parameter fitting focus on the most important,
high-scoring region of a score distribution, and ignore
low-scoring outliers.
\item The collected data are complete, but they are fit to a
distribution that only describes the right (highest scoring)
tail, and the goodness-of-fit test is only performed on that
tail. This case arises when we don't know the form of the
expected distribution for the complete data, but the tail
follows a predictable decay (an exponential tail, for example).
\item The collected data are left-censored such that no values $<
\phi$ were recorded in the histogram, but the data are fit to a
complete distribution that predicts the probability even of the
censored (unobserved) values. Goodness of fit is only evaluated
in the observed data. (This case is what is actually meant by
left-censored data.)
\item The high-scoring right tail of the collected data are fit as the
\emph{binned} counts in the histogram (not raw sample values) to
a distribution that describes the tail, such as an
exponential. This case becomes useful when the raw data values
have limited precision (because of rounding, for example), which
can cause numerical problems with parameter fitting to tails.
Another case where this is useful is when there are so many data
points that the data must be binned just as a matter of
practicality (not enough memory to hold a full histogram).
\end{enumerate}
A variety of other situations can be dealt with by using different
combinations of the function calls that deal with these four cases.
\subsubsection{Focusing parameter fitting on the highest scores}
An example of focusing a Gumbel parameter fit on the right half of an
observed distribution:
\input{cexcerpts/histogram_example2}
The key differences from the complete data case are:
\begin{itemize}
\item Only the high-scoring 50\% of the data samples are
retrieved, by calling
\ccode{esl\_histogram\_GetTailByMass(h, 0.5, \&xv, \&n, \&z)}.
This returns \ccode{z}, the number of samples that
were \emph{censored}.
\item These data are fit to a Gumbel distribution
as a \emph{left-censored} dataset by calling
\ccode{esl\_gumbel\_FitCensored(xv, n, z, xv[0], \&mu, \&lambda)}.
\end{itemize}
The expected counts and the goodness of fit tests are still evaluated
for the complete data, even though the fit was performed only on the
highest scores.
\subsubsection{Fitting to a tail distribution}
An example of fitting an exponential tail to the high-scoring 10\% of
a Gumbel-distributed dataset:
\input{cexcerpts/histogram_example3}
The differences to note are:
\begin{itemize}
\item The tail is fit as if it is \emph{complete} data as far
as the exponential distribution is concerned.
\item As a result, to use the exponential tail to predict expected
data, we have to keep in mind how much probability mass the tail
is supposed to predict (here, 10\%), and that
is provided to
\ccode{esl\_histogram\_SetExpectedTail()}, which specifically
calculates expected counts for a tail.
\end{itemize}
\subsubsection{Fitting left-censored data}
Fitting a Gumbel distribution to data that are \emph{truly} left
censored looks a lot like the case where we extracted the high scoring
data for a censored fit:
\input{cexcerpts/histogram_example4}
\subsubsection{Fitting binned data to a tail distribution}
Normally, you want to fit parameters to the actual individual data
samples, not to binned data, because you'll get more accurate results.
An exception can arise when the data samples have limited precision
because they've been rounded off. Most distributions are not sensitive
to this, but some tail densities are, especially those with
singularities ($P(X=x) \rightarrow \infty$) at their origin. In such a
case, a fit to binned data may be superior, especially if you can
match the histogram's bins to the rounding procedure that was used.
The following code shows an example of fitting for samples that were
already rounded up to the nearest integer before adding them to the
histogram:
\input{cexcerpts/histogram_example5}
Issues to note:
\begin{itemize}
\item The \ccode{esl\_histogram\_Create(-100, 100, 1.0)} call
defined bins that exactly match the rounding procedure
defined by \ccode{ceil(x)} -- all $x$ that are rounded
to the same value by \ccode{ceil(x)} would also go in
the same bin of the histogram.
\item The \ccode{esl\_histogram\_SetTailByMass()} function sets flags
in the histogram to demarcate the desired tail. However,
because the data have been binned, and we can only define the
tail by a range of bins, it will generally be impossible to
match the requested tail mass with adequate accuracy; the actual
tail mass is $\geq$ the requested tail mass. It is returned
to the caller, and it is the actual mass, not the requested mass,
that should be used when setting expected counts.
\item The \ccode{esl\_histogram\_SetRounding()} declaration
sets a flag in the histogram that tells binned parameter
fitting functions that the origin of the fitted
density ($\mu$) should be set at the lower bound of the smallest bin,
rather than the smallest raw data value observed in that
bin.
\end{itemize}
You can’t perform that action at this time.