[REVIEW]: Model dispersion with PRISM; an alternative to MCMC for rapid analysis of models

[whedon]

Submitting author: @1313e (Ellert van der Velden)
Repository: https://github.com/1313e/PRISM
Version: v1.0.0
Editor: @arokem
Reviewer: @fonnesbeck
Archive: 10.5281/zenodo.2572736

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf"><img src="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg)](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@fonnesbeck, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @fonnesbeck

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v1.0.0)?
• Authorship: Has the submitting author (@1313e) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @fonnesbeck it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[1313e]

As mentioned in the blog here (http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing),
it must be clear to all parties involved that a JOSS submission is associated with an AAS Journals submission.
Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@fonnesbeck @arokem Do you need the AAS Journals submission reference for this by any chance?

[arfon]

Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@1313e many thanks for letting us know about this - I think you're the first!

@fonnesbeck - when you get a chance, please review this blog post from late last year: http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing . In particular, please let us know if you have any objection to JOSS receiving a small donation from AAS publishing for this.

[1313e]

@1313e many thanks for letting us know about this - I think you're the first!

Yeah, the news about that collaboration came just in time before my paper submission, so I am not surprised I am the first one.

[1313e]

@arfon May I btw mention that AAS Journals has no system in place at all that a submission is being accompanied by a JOSS submission?
I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

[arfon]

I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

Yes, I think this is still a work in progress.

/ cc @crawfordsm who is the software editor for AAS so he has some visibility into this review too.

[crawfordsm]

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

[1313e]

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

@crawfordsm Do you, by any chance, need the submission reference, to make it easier to find which paper it actually is that I submitted to AAS Journals?

[crawfordsm]

It's okay, I have the information. I am the Scientific Editor for the paper, so I can monitor it here as well as on the AAS side. I realize now that I had just overlooked the information. As of right now, we do not have a formal process for this, but appreciate your patience as we figure out the details.

[arokem]

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

[arfon]

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

[arokem]

Understood! OK - @fonnesbeck - if this all sounds agreeable to you, please do go ahead with your review. Once the review process here is over, we can circle back and see what we need to do. Thanks!

[1313e]

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

@arfon That would be required anyways, given that I am citing the AAS Journals submission for the results.
I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

I hope that is okay btw, as I couldn't find any guidelines on this.

[arfon]

I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

👍 yes that's right. We also don't allow scientific results to be published in JOSS :-)

I hope that is okay btw, as I couldn't find any guidelines on this.

I think you've got this right. @crawfordsm and I are in the process of drafting some guidelines that should assist future authors.

[fonnesbeck]

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

[fonnesbeck]

I'm having problems running the example model. See 1313e/PRISM#7

[1313e]

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

@fonnesbeck You will have to install all requirements in requirements_dev.txt before running the pytests.
I could put that in the README for clarity, if required.

[fonnesbeck]

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

[1313e]

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

@fonnesbeck Alright, I have added it to the README.
I have also "fixed" 1313e/PRISM#7.

[fonnesbeck]

Tests now run after installing the dev requirements, but the tests fail.

[fonnesbeck]

The submission satisfies all of the evaluation criteria. Once the issue above is addressed, I'm happy to recommend ACCEPT.

[1313e]

@fonnesbeck I have solved the problem with 1313e/PRISM#8, and also improved the README and docs on the topic of running the pytests.

@arokem Do you have an idea how I can fix the formatting error that whedon gives when compiling the paper.md?
It formats my last name incorrectly in the footer of every page (I already noticed that immediately after submission, but had not asked about it yet).

[arfon]

@1313e - I’ll have to fix the name issue manually at the end (the automated proofs will continue to have this problem)

[fonnesbeck]

I am now able to run the test suite without issue. There are no other outstanding problems, so I recommend ACCEPT. cc @arfon

[1313e]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[arfon]

@1313e - I'm going to go ahead and accept this paper now as I believe the AAS paper is very close to being published too.

Apologies for the back and forth on this submission - you're the first author to follow the dual-publishing workflow in AAS/JOSS and we've been finding (and fixing) a few challenges along the way. Your patience has been much appreciated!

[arfon]

@fonnesbeck - many thanks for your review here and to @arokem for editing this submission ✨

@1313e - your paper is now accepted into JOSS ⚡🚀💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01229/status.svg)](https://doi.org/10.21105/joss.01229)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01229">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01229/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01229/status.svg
   :target: https://doi.org/10.21105/joss.01229

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss

[1313e]

@arfon Will you also make sure that my last name is properly written in the published version?

[arfon]

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

[1313e]

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

Looks fine to me.

[1313e]

@arfon I was actually just looking at the paper, and noticed that the citations are not displayed properly. The first in every sequence is, but the other ones are not.
As far as I know, this was not the case with the automatically compiled versions.
Did I write the citations wrong?

[arfon]

@1313e - could you be more specific about what you're seeing that you believe is incorrect? i.e. give me an example of expected vs actual?

[1313e]

@arfon Well, the second sentence of the third paragraph shows up as "PRISM uses the Bayes linear approach (Goldstein & Wooff, 2007), emulation technique (Craig, Goldstein, Seheult, & Smith, 1996, p. @craig97) and history matching (Raftery, Givens, & Zeh, 1995, pp. @Craig96, @craig97)...".
I have the feeling that a few citations here were not actually formatted.

[arfon]

Ah, right. Sorry about that. The correct syntax for multiple citations is to separate them with semi-colons (e.g. [@Raftery95; @Craig96; @Craig97]). I've fixed up the paper locally in https://github.com/openjournals/joss-papers/blob/a66fbee618680adc4825d723edcbb40165e1dda0/joss.01229/10.21105.joss.01229.pdf and it should be correct now on the live site (note it can take a few hours for the updated PDF to show properly because of caching on our website).

[1313e]

@arfon Thanks for that, but I think you will have to manually set my last name in the footer again...
Similarly, I think that the BibTeX entry for the paper will also have my name incorrectly (as that is the case for the Hickle paper, published here in JOSS).
Is there anything on my part that I can do to avoid that?

[arfon]

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

[1313e]

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

That link does not resolve.

[arfon]

Weird works fine for me. Can you try going to https://www.doi2bib.org/ and putting in 10.21105/joss.01229.

If that doesn't work, the output from the service (i.e. the bibtex entry from the Crossref metadata) is:

@article{vanderVelden2019,
  doi = {10.21105/joss.01229},
  url = {https://doi.org/10.21105/joss.01229},
  year = {2019},
  month = jun,
  publisher = {The Open Journal},
  volume = {4},
  number = {38},
  pages = {1229},
  author = {Ellert van der Velden},
  title = {Model dispersion with {PRISM}$\mathsemicolon$ an alternative to {MCMC} for rapid analysis of models},
  journal = {Journal of Open Source Software}
}

☝️does that look OK?

[1313e]

Uhm, my last name should be in braces, as otherwise it will not be formatted correctly.
Besides that, it looks good.

It does seem though that for my paper, my name is at least written correctly.
The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

[arfon]

OK, thanks. I don't have any control over where https://www.doi2bib.org puts the braces but the metadata looks correct.

The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

OK, that's fixed for 10.21105/joss.01115 in https://github.com/openjournals/joss-papers/blob/master/joss.01115/10.21105.joss.01115.crossref.xml#L43-L47 but it will take a few hours to update on https://www.doi2bib.org .

[1313e]

Alright, thanks.
I know from experience that having no braces can sometimes mess it up when using natbib for example, but I guess that if the metadata is correct, such a thing would not be hard to notice.

[1313e]

@arfon This is not that big of a deal, but I just realized that the footer in the JOSS paper says 'van der Velden et al.', even though I am the sole author on the paper.

[arfon]

@arfon This is not that big of a deal, but I just realized that the footer in the JOSS paper says 'van der Velden et al.', even though I am the sole author on the paper.

Sorry about that. This should be fixed now. Note, the new pdf can take a few hours to show up sometimes because of browser caching.