Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: pymcmcstat: A Python Package for Bayesian Inference Using Delayed Rejection Adaptive Metropolis #1417

Closed
whedon opened this issue Apr 29, 2019 · 82 comments

Comments

Projects
None yet
6 participants
@whedon
Copy link
Collaborator

commented Apr 29, 2019

Submitting author: @prmiles (Paul Miles)
Repository: https://github.com/prmiles/pymcmcstat
Version: v1.7.1
Editor: @katyhuff
Reviewers: @djmitche, @katyhuff
Archive: 10.5281/zenodo.2660145

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/5ac18236dc3f11e9f46977fc43f61991"><img src="http://joss.theoj.org/papers/5ac18236dc3f11e9f46977fc43f61991/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/5ac18236dc3f11e9f46977fc43f61991/status.svg)](http://joss.theoj.org/papers/5ac18236dc3f11e9f46977fc43f61991)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@djmitche & @jarvist, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @katyhuff know.

Please try and complete your review in the next two weeks

Review checklist for @djmitche

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: v1.7.1
  • Authorship: Has the submitting author (@prmiles) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @jarvist

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: v1.7.1
  • Authorship: Has the submitting author (@prmiles) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 29, 2019

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @djmitche, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 29, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 29, 2019

@djmitche

This comment has been minimized.

Copy link

commented May 2, 2019

I'm happy aside from the above-referenced two issues.

While this does not have a very "pythonic" interface, that's justified in the documentation as an effort to match the MATLAB functionality, and I think will match the expectations of users of this sort of application nicely. In particular, I'd highlight that many methods "display" data by printing to stdout rather than return it, and that running the simulation modifies the MCMC object itself, from which results must be fetched, rather than leaving the object unchanged and returning the simulation results. It seems that the displayed results are also available programmatically if necessary (e.g., ModelSettings), so this would not be an impediment to using the library as a component of a larger application.

@katyhuff

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Thank you @djmitche for your prompt review! You've made some very good points that perhaps @prmiles could consider as improvements to the package!

@prmiles

This comment has been minimized.

Copy link

commented May 3, 2019

Thank you @djmitche for your comments regarding the pymcmcstat package. I have resolved the two issues you reported.

@djmitche

This comment has been minimized.

Copy link

commented May 6, 2019

Indeed! All boxes checked..

@djmitche djmitche removed their assignment May 12, 2019

@djmitche

This comment has been minimized.

Copy link

commented May 13, 2019

@jarvist ping!

@katyhuff

This comment has been minimized.

Copy link
Member

commented May 13, 2019

@jarvist : Do you still intend to complete this review?

@katyhuff

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Sorry for the delay here @prmiles. I've to the prompt and thorough review by @djmitche Without the review from @jarvist, I'll move forward with this review on my own, as I have previously installed and checked the software myself. I'll review all the checks today and we can move forward.

Review checklist for @katyhuff

Conflict of interest

  • As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work. Note: I am aware of the authors based on our participation in a large national consortium. We do not have any collaborative work, however.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v1.7.0)?
  • Authorship: Has the submitting author (@prmiles) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@katyhuff

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 4, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 4, 2019

@katyhuff

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@whedon check references

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 4, 2019

Attempting to check references...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 4, 2019


OK DOIs

- 10.1007/s11222-008-9110-y is OK
- 10.1007/s11222-006-9438-0 is OK
- 10.1198/jcgs.2009.06134 is OK
- 10.5281/zenodo.2630882 is OK

MISSING DOIs

- None

INVALID DOIs

- None
@katyhuff

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Thank you @djmitche for your review -- we couldn't do this without you.

Thank you @prmiles for a strong submission and for engaging actively in the review process! I have looked over the paper, double checked all the DOI links, and have conducted a high level review of the code itself. Everything looks ship-shape to me. At this point, please double check the paper yourself, review any lingering details in your code/readme/etc. that you intend to make, and then make an archive of the reviewed software in Zenodo/figshare/other service. Please be sure that the DOI metadata (title, authors, etc.) matches this JOSS submission. You may update the version to reflect these changes as necessary. If you do need to update the version please let me know the new version number. Once that's complete, please update this thread with the DOI of the archive, and I'll move forward with accepting the submission! Until then, now is your moment for final touchups!

@katyhuff

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@whedon remove @jarvist as reviewer

@prmiles

This comment has been minimized.

Copy link

commented Jun 5, 2019

@katyhuff Thank you for the suggestion regarding the 4th paragraph! I'm grateful the issue was brought up, and thank you for your patience as I make the updates.

@djmitche

This comment has been minimized.

Copy link

commented Jun 5, 2019

I'll look more closely at that in future reviews. I did have to do a bit of background reading to understand the purpose of the package.

@prmiles

This comment has been minimized.

Copy link

commented Jun 6, 2019

I have modified the paper to accommodate a more general technical audience. The first couple paragraphs provide a broad perspective of why the software is useful and where it can be used in scientific research (hopefully!). My hope is that the first part of the paper gives the reader the big picture and enough information to know whether or not they could use the software.

The latter part of the paper still contains fairly technical content. I added some discussion regarding Metropolis algorithms, and I believe the references provide a reasonable set of resources if someone wants to learn more about MCMC methods.

Ideally, the first part of the paper makes the technical content less overwhelming, but I am happy to continue iterating. Thank you again for the feedback regarding the paper!

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

Please delete the last sentence before the acknowledgements, and the citation to the Zenodo deposit. We ask that you don't include this as a citation, because the archive will be highlighted in the margin decorators on the first page of the paper (and the DOIs will be linked in the metadata of the paper).

@prmiles

This comment has been minimized.

Copy link

commented Jun 6, 2019

I have removed the final sentence, including the citation to the Zenodo deposit. I also removed the Zenodo information from the .bib file.

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@whedon accept

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Attempting dry run of processing paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019


OK DOIs

- 10.1007/s11222-008-9110-y is OK
- 10.1007/s11222-006-9438-0 is OK
- 10.1198/jcgs.2009.06134 is OK
- 10.5281/zenodo.2660145 is OK

MISSING DOIs

- None

INVALID DOIs

- None
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Check final proof 👉 openjournals/joss-papers#738

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#738, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

Hmm. Did you forget to push the change? I still see the sentence and citation in the PDF created with the accept command above.

@prmiles

This comment has been minimized.

Copy link

commented Jun 6, 2019

The paper looks correct in the master version of my GitHub repository... I'm not entirely sure how the compiling process works with whedon. Does whedon pull the latest version from the repository when you use the accept command? Or only when you generate the pdf?

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@whedon accept

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Attempting dry run of processing paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019


OK DOIs

- 10.1007/s11222-008-9110-y is OK
- 10.1007/s11222-006-9438-0 is OK
- 10.1198/jcgs.2009.06134 is OK

MISSING DOIs

- None

INVALID DOIs

- None
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Check final proof 👉 openjournals/joss-papers#739

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#739, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

@whedon accept deposit=true

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

Doing it live! Attempting automated processing of paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

  1. Check final PDF and Crossref metadata that was deposited 👉 openjournals/joss-papers#740
  2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01417
  3. If everything looks good, then close this review issue.
  4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

Congrats, @prmiles, your paper is published!
Huge thanks to our Rditor: @katyhuff, and reviewers: @djmitche, @katyhuff

@labarba

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

(Published from my phone at an airport)

@labarba labarba closed this Jun 6, 2019

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 6, 2019

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01417/status.svg)](https://doi.org/10.21105/joss.01417)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01417">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01417/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01417/status.svg
   :target: https://doi.org/10.21105/joss.01417

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

@prmiles

This comment has been minimized.

Copy link

commented Jun 7, 2019

Thank you again for all the feedback on the paper and software!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.