Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/improve nuts diagnostic #352

Merged
merged 6 commits into from
Nov 26, 2013
Merged

Feature/improve nuts diagnostic #352

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2ae82f8
Tweak NUTS acceptance statistic, introduce number of divergent iterat…
betanalpha Nov 1, 2013
b88cf01
Merge branch 'develop' into feature/improve_nuts_diagnostic
betanalpha Nov 2, 2013
f226209
Update sampler param name to be consistent with doc
betanalpha Nov 11, 2013
bc55916
Update sampler param name to be consistent with doc
betanalpha Nov 11, 2013
4e1cfcf
Update model tests to ignore new diagnostic
betanalpha Nov 14, 2013
964b518
Remove last reference to n_divergence
betanalpha Nov 21, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions src/docs/stan-reference/commands.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2147,8 +2147,9 @@ \section{Running the Print Command}
accept_stat__ 0.88 5.6e-03 1.8e-01 0.51 0.95 1.0 1000 16881 1.00
stepsize__ 0.30 1.3e-15 8.9e-16 0.30 0.30 0.30 0.50 8.5 1.00
treedepth__ 1.4 2.6e-02 8.0e-01 0.00 1.0 2.0 946 15978 1.00
n_divergent__ 1.4 0.0e+00 0.0e+00 0.00 0.0 0.0 1000 16949 1.00
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be n_divergence__?

alpha 17 1.8e+00 2.5e+01 1.9 9.5 50 181 3054 1.00
beta 10 1.1e+00 1.4e+01 1.2 6.2 31 181 3057 1.0
beta 10 1.1e+00 1.4e+01 1.2 6.2 31 181 3057 1.00

Samples were drawn using hmc with nuts.
For each parameter, N_Eff is a crude measure of effective sample size,
Expand All @@ -2168,7 +2169,10 @@ \section{Running the Print Command}
step size used by NUTS in its Hamiltonian simulation, and
\code{treedepth\_\_} is the depth of tree used by NUTS, which is the
log (base 2) of the number of leapfrog steps taken during the
Hamiltonian simulation. }\label{bin-print-eg.figure}
Hamiltonian simulation. \code{n\_divergent\_\_} gives the number
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n_diverence__?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used "divergent" as an adjective to the iteration "as in divergent iteration", "not divergence iteration" but I'm happy to switch it if people would prefer a different notation.

On Nov 11, 2013, at 9:59 PM, Daniel Lee notifications@github.com wrote:

In src/docs/stan-reference/commands.tex:

@@ -2168,7 +2169,10 @@ \section{Running the Print Command}
step size used by NUTS in its Hamiltonian simulation, and
\code{treedepth__} is the depth of tree used by NUTS, which is the
log (base 2) of the number of leapfrog steps taken during the

  • Hamiltonian simulation. }\label{bin-print-eg.figure}
  • Hamiltonian simulation. \code{n_divergent__} gives the number

n_diverence__?

—
Reply to this email directly or view it on GitHub.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just looking at consistent usage. The code (or at least the tests you have) indicates that n_divergence__ is printed in the header files of output and in the output of bin/print, not n_divergent__.

Feel free to go with one or the other. It's ok to use "divergent" as an adjective too, but if the code is generating n_divergence__, I'd prefer that the doc is consistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, misread that. Branch updated to be consistent.

On Nov 11, 2013, at 10:38 PM, Daniel Lee notifications@github.com wrote:

In src/docs/stan-reference/commands.tex:

@@ -2168,7 +2169,10 @@ \section{Running the Print Command}
step size used by NUTS in its Hamiltonian simulation, and
\code{treedepth__} is the depth of tree used by NUTS, which is the
log (base 2) of the number of leapfrog steps taken during the

  • Hamiltonian simulation. }\label{bin-print-eg.figure}
  • Hamiltonian simulation. \code{n_divergent__} gives the number

I was just looking at consistent usage. The code (or at least the tests you have) indicates that n_divergence__ is printed in the header files of output and in the output of bin/print, not n_divergent__.

Feel free to go with one or the other. It's ok to use "divergent" as an adjective too, but if the code is generating n_divergence__, I'd prefer that the doc is consistent.

—
Reply to this email directly or view it on GitHub.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for "divergent"

On 11/11/13, 5:19 PM, Michael Betancourt wrote:

In src/docs/stan-reference/commands.tex:

@@ -2168,7 +2169,10 @@ \section{Running the Print Command}
step size used by NUTS in its Hamiltonian simulation, and
\code{treedepth__} is the depth of tree used by NUTS, which is the
log (base 2) of the number of leapfrog steps taken during the

  • Hamiltonian simulation. }\label{bin-print-eg.figure}
  • Hamiltonian simulation. \code{n_divergent__} gives the number

I used "divergent" as an adjective to the iteration "as in divergent iteration", "not divergence iteration" but I'm
happy to switch it if people would prefer a different notation.
… <#>
On Nov 11, 2013, at 9:59 PM, Daniel Lee notifications@github.com wrote: In src/docs/stan-reference/commands.tex: > @@
-2168,7 +2169,10 @@ \section{Running the Print Command} > step size used by NUTS in its Hamiltonian simulation, and >
\code{treedepth__} is the depth of tree used by NUTS, which is the > log (base 2) of the number of leapfrog steps
taken during the > - Hamiltonian simulation. }\label{bin-print-eg.figure} > + Hamiltonian simulation.
\code{n_divergent__} gives the number n_diverence__? — Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub https://github.com/stan-dev/stan/pull/352/files#r7575676.

of leapfrog iterations with diverging error; because NUTS terminates
at the first divergent iteration this should always be either 0 or 1.}
\label{bin-print-eg.figure}
\end{figure}
%\end{quote}
%
Expand Down
45 changes: 24 additions & 21 deletions src/docs/stan-reference/introduction.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -603,7 +603,7 @@ \subsubsection{Sampler Output}
\begin{quote}
\begin{Verbatim}[fontsize=\small]
...
lp__,accept_stat__,stepsize__,treedepth__,theta
lp__,accept_stat__,stepsize__,treedepth__,n_divergent__,theta
...
\end{Verbatim}
\end{quote}
Expand All @@ -630,6 +630,8 @@ \subsubsection{Sampler Output}
\\
\NUTS & \Verb| treedepth__ | & Tree depth
\\
\NUTS & \Verb| n_divergent__ | & Number of divergent iterations
\\
\end{tabular}
\end{center}

Expand Down Expand Up @@ -661,27 +663,27 @@ \subsubsection{Sampler Output}
\begin{quote}
\begin{Verbatim}[fontsize=\small]
...
-6.85935,1,1.81311,0,0.194323
-6.85935,0.0001295,1.81311,0,0.194323
-6.79366,1,1.81311,0,0.288971
-6.77936,1,1.81311,0,0.282131
-6.77936,0.886654,1.81311,0,0.282131
-6.74812,1,1.81311,0,0.248232
-6.74812,0.0793306,1.81311,0,0.248232
-7.04438,0.777304,1.81311,0,0.353339
-7.04438,0.480879,1.81311,0,0.353339
-7.07619,0.849599,1.81311,0,0.158857
-6.85935,1,1.81311,0,0,0.194323
-6.85935,0.0001295,1.81311,0,0,0.194323
-6.79366,1,1.81311,0,0,0.288971
-6.77936,1,1.81311,0,0,0.282131
-6.77936,0.886654,1.81311,0,0,0.282131
-6.74812,1,1.81311,0,0,0.248232
-6.74812,0.0793306,1.81311,0,0,0.248232
-7.04438,0.777304,1.81311,0,0,0.353339
-7.04438,0.480879,1.81311,0,0,0.353339
-7.07619,0.849599,1.81311,0,0,0.158857
...
-6.74838,0.540129,1.81311,0,0.253368
-6.74838,0.638693,1.81311,0,0.253368
-6.74838,0.546,1.81311,0,0.253368
-7.01801,0.794591,1.81311,0,0.348376
-7.01801,0.0564743,1.81311,0,0.348376
-7.17267,0.863864,1.81311,0,0.375023
-6.86632,1,1.81311,0,0.192719
-6.86632,0.0077144,1.81311,0,0.192719
-8.21853,0.512368,1.81311,0,0.084075
-8.7342,0.862726,1.81311,0,0.0669754
-6.74838,0.540129,1.81311,0,0,0.253368
-6.74838,0.638693,1.81311,0,0,0.253368
-6.74838,0.546,1.81311,0,0,0.253368
-7.01801,0.794591,1.81311,0,0,0.348376
-7.01801,0.0564743,1.81311,0,0,0.348376
-7.17267,0.863864,1.81311,0,0,0.375023
-6.86632,1,1.81311,0,0,0.192719
-6.86632,0.0077144,1.81311,0,0,0.192719
-8.21853,0.512368,1.81311,0,0,0.084075
-8.7342,0.862726,1.81311,0,0,0.0669754
...
\end{Verbatim}
\end{quote}
Expand Down Expand Up @@ -735,6 +737,7 @@ \subsubsection{Summarizing Sampler Output}
accept_stat__ 0.64 1.2e-02 3.6e-01 5.1e-03 0.74 1.0 882 76898 1.00
stepsize__ 1.8 7.8e-15 5.6e-15 1.8e+00 1.8 1.8 0.50 44 1.00
treedepth__ 0.076 8.6e-03 2.7e-01 0.0e+00 0.00 1.0 942 82167 1.00
n_divergence__ 0.00 0.0e+00 0.0e+00 0.0e+00 0.00 0.00 1000 90909 1.00
theta 0.25 4.2e-03 1.2e-01 9.0e-02 0.23 0.47 827 72146 1.00

Samples were drawn using hmc with nuts.
Expand Down
20 changes: 13 additions & 7 deletions src/stan/mcmc/hmc/nuts/base_nuts.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ namespace stan {

base_nuts(M &m, BaseRNG& rng, std::ostream* o, std::ostream* e):
base_hmc<M, P, H, I, BaseRNG>(m, rng, o, e),
_depth(0), _max_depth(5), _max_delta(1000)
_depth(0), _max_depth(5), _max_delta(1000), _n_divergence(0)
{};

~base_nuts() {};
Expand Down Expand Up @@ -88,12 +88,13 @@ namespace stan {
int n_valid = 0;

this->_depth = 0;
this->_n_divergence = 0;

util.n_tree = 0;
util.sum_prob = 0;

while (util.criterion && (this->_depth <= this->_max_depth) ) {

util.n_tree = 0;
util.sum_prob = 0;

// Randomly sample a direction in time
ps_point* z = 0;
Eigen::VectorXd* rho = 0;
Expand Down Expand Up @@ -156,21 +157,23 @@ namespace stan {
}

void write_sampler_param_names(std::ostream& o) {
o << "stepsize__,treedepth__,";
o << "stepsize__,treedepth__,n_divergence__,";
}

void write_sampler_params(std::ostream& o) {
o << this->_epsilon << "," << this->_depth << ",";
o << this->_epsilon << "," << this->_depth << "," << this->_n_divergence << ",";
}

void get_sampler_param_names(std::vector<std::string>& names) {
names.push_back("stepsize__");
names.push_back("treedepth__");
names.push_back("n_divergence__");
}

void get_sampler_params(std::vector<double>& values) {
values.push_back(this->_epsilon);
values.push_back(this->_depth);
values.push_back(this->_n_divergence);
}

virtual bool compute_criterion(ps_point& start, P& finish, Eigen::VectorXd& rho) = 0;
Expand All @@ -197,7 +200,8 @@ namespace stan {
if (boost::math::isnan(h)) h = std::numeric_limits<double>::infinity();

util.criterion = util.log_u + (h - util.H0) < this->_max_delta;

if (!util.criterion) ++(this->_n_divergence);

util.sum_prob += stan::math::min(1, std::exp(util.H0 - h));
util.n_tree += 1;

Expand Down Expand Up @@ -244,6 +248,8 @@ namespace stan {
int _depth;
int _max_depth;
double _max_delta;

int _n_divergence;

};

Expand Down
6 changes: 4 additions & 2 deletions src/test/io/mcmc_writer_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ TEST(StanIoMcmcWriter, print_sample_names) {
std::string line;
std::getline(sample_stream, line);

EXPECT_EQ("lp__,accept_stat__,stepsize__,treedepth__,mu1,mu2", line);
EXPECT_EQ("lp__,accept_stat__,stepsize__,treedepth__,n_divergence__,mu1,mu2", line);

}

Expand Down Expand Up @@ -100,6 +100,7 @@ TEST(StanIoMcmcWriter, print_sample_params) {
expected_stream << accept_stat << ",";
expected_stream << sampler.get_current_stepsize() << ",";
expected_stream << 0 << ",";
expected_stream << 0 << ",";
expected_stream << real.at(0) << ",";
expected_stream << real.at(1);

Expand Down Expand Up @@ -233,7 +234,7 @@ TEST(StanIoMcmcWriter, print_diagnostic_names) {
std::getline(diagnostic_stream, line);

// FIXME: make this work, too
EXPECT_EQ("lp__,accept_stat__,stepsize__,treedepth__,mu1,mu2,p_mu1,p_mu2,g_mu1,g_mu2", line);
EXPECT_EQ("lp__,accept_stat__,stepsize__,treedepth__,n_divergence__,mu1,mu2,p_mu1,p_mu2,g_mu1,g_mu2", line);

}

Expand Down Expand Up @@ -285,6 +286,7 @@ TEST(StanIoMcmcWriter, print_diagnostic_params) {
expected_stream << accept_stat << ",";
expected_stream << sampler.get_current_stepsize() << ",";
expected_stream << 0 << ",";
expected_stream << 0 << ",";
expected_stream << real.at(0) << ",";
expected_stream << real.at(1) << ",";
expected_stream << 0 << ",";
Expand Down