-
-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/1263 update stansummary to report rank-normalized ESS tail, ESS bulk, max abs deviation(MAD), and Rhat #1290
base: develop
Are you sure you want to change the base?
Conversation
…ature/1263-new-rhat-summary
This PR is a WIP and will fail until Stan PR stan-dev/stan#3305 is merged. |
Jenkins Console Log Machine informationNo LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focalCPU: G++: Clang: |
…an-dev/cmdstan into feature/1263-new-rhat-summary
@@ -702,14 +558,17 @@ void write_sampler_info(const stan::io::stan_csv_metadata &metadata, | |||
*out << prefix << "Samples were drawn using " << metadata.algorithm | |||
<< " with " << metadata.engine << "." << std::endl; | |||
*out << prefix | |||
<< "For each parameter, N_Eff is a crude measure of effective " | |||
"sample size," | |||
<< "For each parameter, N_Eff_bulk and N_Eff_tail measure the " |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Want to tack #916 on to this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, definitely!
src/cmdstan/stansummary_helper.hpp
Outdated
header.at(0) = "Mean"; | ||
header.at(1) = "MCSE"; | ||
header.at(2) = "StdDev"; | ||
header.at(2) = "MaxAbsDev"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand the rhat/ess changes are mostly replacements, but do we really not want to report standard deviations any more?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
excellent question. do both? should it be "MaxAbsDev" or "MAD" ? I had to look up MAD in the R ecosystem docs; it's a different view on how spread out the posterior distribution is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m not familiar with MAD, but both always seems like a safe option
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The text explaining ESS and Rhat could be updated to include descriptions of MCSE and MAD.
changes to stansummary are running afoul of this PR: #972 |
Jenkins Console Log Machine informationNo LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focalCPU: G++: Clang: |
Jenkins Console Log Machine informationNo LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focalCPU: G++: Clang: |
…an-dev/cmdstan into feature/1263-new-rhat-summary
Submisison Checklist
./runCmdStanTests.py src/test
Summary:
Convergence diagnostics reported by
stansummary
utility use split-rank-normalized Rhat and ESS, following paper https://arxiv.org/pdf/1903.08008.Intended Effect:
Better convergence diagnostics.
How to Verify:
Unit tests
Side Effects:
Output of
bin/stansummary
will have different set of columns, and slightly different messages.Documentation:
Separate PR
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses: