-
-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/2569 Analysis API: compute effective sample size #2575
Feature/2569 Analysis API: compute effective sample size #2575
Conversation
Please note my use of a function template on CmdStan is able/will instantiate RStan is able/will instantiate either the above or the below signature, as I don't think Rcpp is as unhappy about Is PyStan able to instantiate @ariddell, would you please opine your ability to deal with my use of a function template? |
That signature looks fine to me. I don't think there should be any problem calling that function with Cython from PyStan. |
10/11 travis checks were successful. The one error just says no output received in the last 10 minutes. I can't find more details on the error. Let me know what I can do to help. |
Thanks for the heads up. I restarted the job. If you don't have credentials for kicking Travis, @seantalts should be able to set you up. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The biggest thing to change here is the implementation of chains. It shouldn't have picked up a new method. Instead, the old method should have dispatched to your new function.
* @param std::vector stores sizes of chains | ||
* @return effective sample size for the specified parameter | ||
*/ | ||
template<typename T> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this templated? It looks like on the issue and looking at the implementation, this can be removed and the type of the input be double *
. I don't know if it needs to be templated for something like RStan to use well, but if so, please document it.
If leaving the template parameter in, please add the template parameter documentation using @tparam
. See: https://github.com/stan-dev/stan/wiki/How-to-Write-Doxygen-Doc-Comments#function-doc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for asking, as I was hoping for feedback on this. I addressed my reasoning behind templating this function in a comment immediately following this PR's initial comment. I've come to learn that you'd prefer such details/requests for feedback in the PR's initial comment.
The short story is that CmdStan won't have to make any copies if it can use compute_effective_sample_size(std::vector<const double*>, std::vector<size_t> sizes)
, but everyone agreed to compute_effective_sample_size(std::vector<double*>, std::vector<size_t> sizes)
, for which CmdStan will necessitate a copy. In an attempt to satisfy all requests of our previous discussions, I templated this function.
If RStan and PyStan can both call either signature, then I'd recommend we change the type of the first argument from std::vector<double*>
to std::vector<const double*>
and drop the template.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for bringing this up! We should use const double *
. I think it should actually be const double * const
. The pointers don't change and we never mutate the elements.
CmdStan should be able to handle it. We'll figure out how to make RStan and PyStan work with that signature, even if it means casting the consts away.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for bringing this up! We should use const double *
. I think it should actually be const double * const
. The pointers don't change and we never mutate the elements.
CmdStan should be able to handle it. We'll figure out how to make RStan and PyStan work with that signature, even if it means casting the consts away.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The const double * const
is a good thought and I agree with the reasoning behind it, but I don't know how to deal with that type here. Wouldn't this necessitate initialization at declaration, since it won't allow modification later? For instance, I don't see a reasonable way get the rvalues of line 616 in src/stan/mcmc/chains.hpp into the (would be) container std::vector<const double * const> draws
.
Eigen::VectorXd chain_mean(num_chains); | ||
Eigen::VectorXd chain_var(num_chains); | ||
for (size_t chain = 0; chain < num_chains; ++chain) { | ||
Eigen::Map<const Eigen::Matrix<double, Eigen::Dynamic, 1> > |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With C++11, we don't need to have the space between the two right angle brackets (> >
). Please remove the space between the two.
@@ -0,0 +1,100 @@ | |||
#ifndef STAN_ANALYZE_ESS_HPP |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please rename the file to match the function name: compute_effective_sample_size.hpp
.
Please update the header guard to match the file structure exactly: STAN_ANALYZE_MCMC_COMPUTE_EFFECTIVE_SAMPLE_SIZE_HPP
.
template<typename T> | ||
double compute_effective_sample_size(std::vector<T> draws, size_t size) { | ||
size_t num_chains = draws.size(); | ||
std::vector<size_t> sizes(num_chains); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a lot easier with C++11! This can just be:
std::vector<size_t> sizes(num_chains, size);
and you can remove the loop!
src/stan/mcmc/chains.hpp
Outdated
@@ -701,6 +702,20 @@ namespace stan { | |||
return effective_sample_size(index(name)); | |||
} | |||
|
|||
double compute_effective_sample_size(const int index) const { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So... this isn't what should happen. Remove this function signature. Replace the existing class method, effective_sample_size()
(line 230), with a dispatch to the function you've written.
We'll replace each part of this class in that fashion where all the work is done outside. Then remove the class itself later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cleaned up chains.hpp as requested.
Should I also remove the effective sample size tests in src/test/unit/mcmc/chain_test.cpp
, so as to rely on the tests in src/test/unit/analyze/mcmc/compute_effective_sample_size.cpp
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be ideal! Thanks! The test in chains should just check that it's callable, not really check for the validity of the value that's returned.
If you don't want to do that, no prob.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be ideal! Thanks! The test in chains should just check that it's callable, not really check for the validity of the value that's returned.
If you don't want to do that, no prob.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RE check that it's callable. Is something like this satisfactory within src/test/unit/mcmc/chains_test.cpp
?
EXPECT_NO_THROW(chains.effective_sample_size(1.0))
<< "calling chain.effective_sample_size(index = 1.0).";
Thanks for the feedback. Please give me some time to address your comments as I don’t have access to a computer until Tuesday. |
No problem! Sorry for the delay on getting it reviewed. Hopefully we can keep up with the queue a little faster. |
…re/issue-2569-analysis-api-ess
…re/issue-2569-analysis-api-ess
Yup! Except that should be the index 1 instead of 1.0? If there's some
error condition it should trigger (index < 0 or greater than the max)
that's currently tested, that's should be kept around. If it's not there,
it's ok not to add it.
…On Tue, Jul 31, 2018 at 5:47 PM Edward A. Roualdes ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/stan/mcmc/chains.hpp
<#2575 (comment)>:
> @@ -701,6 +702,20 @@ namespace stan {
return effective_sample_size(index(name));
}
+ double compute_effective_sample_size(const int index) const {
RE check that it's callable. Is something like this satisfactory within
src/test/unit/mcmc/chains_test.cpp?
EXPECT_NO_THROW(chains.effective_sample_size(1.0))
<< "calling chain.effective_sample_size(index = 1.0).";
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2575 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAZ_F-7RzxUQU_fmjZpsLeWFfYev92sSks5uMNBJgaJpZM4VIPkB>
.
|
…ream rather than my fork
…n's method effective_sample_size
I'm trying to debug this error from jenkins, but failing. The error message on jenkins provides enough for me to locate the error -- something following
but not enough info to move from there. I would like to recreate the above command relative to this branch. When I execute the following, this branch isn't available.
Is this branch not available because it's coming from my fork of stan-dev/stan? Are my attempts at git off base? Is this idea possible? @syclik Any other tips or suggestions you might have to help me move forward would be much appreciated. Thanks. |
After a series of clicks (which I can’t remember), I got to this:
http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/CmdStan/detail/downstream_tests/47/pipeline
Does that help?
…On Wed, Aug 1, 2018 at 11:16 PM Edward A. Roualdes ***@***.***> wrote:
I'm trying to debug this error from jenkins, but failing. The error
message on jenkins provides enough for me to locate the error -- something
following
bin/stansummary src/test/interface/matrix_output.csv
but not enough info to move from there. I would like to recreate the above
command relative to this branch. When I execute the following, this branch
isn't available.
git clone --recursive https://github.com/stan-dev/cmdstan.git
cd stan
# git checkout feature/issue-2569-analysis-api-ess # branch not available
Is this branch not available because it's coming from my fork of
stan-dev/stan? Are my attempts at git off base? Is this idea possible?
@syclik <https://github.com/syclik> Any other tips or suggestions you
might have to help me move forward would be much appreciated. Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2575 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAZ_F7-lJj2r8aeu4UK8PU4DNfux0I5-ks5uMm70gaJpZM4VIPkB>
.
|
When you click on the failing build from github, you get to this page. On that page you can see that the upstream tests failed. At the bottom of the page is a link to the failing upstream tests, awkwardly named "downstream tests" (the red X should draw your attention). Clicking that brings you here, which is where Daniel's link goes. Looks like the interface tests failed on all platforms (we should maybe stage those so they aren't all executing in parallel, haha). |
Thanks to you both for your quick responses. I (previously) made it as far you both suggest. The error comes when the test, named CommandStansummary.functional_test__issue_342, issues the following command: As I see it, either my code isn't building correctly and thus CmdStan can't find the function In an effort to learn why this fails, I'd like to run the same CmdStan command on my personal machine: If this thought process is off target, please don't hesitate to say so. |
I see, sorry. There are failing tests and that's why the command quit with an error. re: testing CmdStan with your branch - Cmdstan has Stan and Math as submodules. Clone your fork of cmdstan with --recursive and then |
That did it. Thanks @seantalts for the tip about remotes. I'll report back when I have something to offer about this error. |
The error: I had previously used The fix: replace Note that neither @syclik Before this gets final approval, did you see my comment about the types |
Sorry -- I missed it the first time around. Here's the last comment:
Please ignore the Thanks! |
Thanks all for your help on this. I especially appreciate your patience with me as I learn and gain experience in Stan and C++11. |
On Aug 3, 2018, at 6:20 PM, Edward A. Roualdes ***@***.***> wrote:
Thanks all for your help on this. I especially appreciate your patience with me as I learn and gain experience in Stan and C++11.
We try to prioritize helping new devs get across the first merged PR hurdle.
After that, you're in the same boat as the rest of us trying to keep up with the moving C++ standard and best practices; we all need to help each other on that.
|
Submission Checklist
./runTests.py src/test/unit
make cpplint
Ran tests mentioned below and
./runTests.py src/test/unit/mcmc
Summary
Create new route for unified calculation of effective sample size. Discussion on Discourse and in issue #2569.
Add function
src/stan/analyze/mcmc/compute_ess.hpp
along with tests.Add method on chains class in
src/stan/mcmc/chains.hpp
in an effort to minimize changes necessary for adoption into CmdStan.Intended Effect
Add unified API for interfaces to compute effective sample size, and maintain backwards compatibility during the transition to the new route.
How to Verify
Tests live in
src/stan/test/unit/analyze/mcmc/compte_ess_test.cpp
. Runmake clean && ./runTests.py -j2 src/test/unit/analyze/
Side Effects
None intended.
Documentation
Inline, via doxygen.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): California State University, Chico
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses: