Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: Model dispersion with PRISM; an alternative to MCMC for rapid analysis of models #1229

Closed
whedon opened this issue Feb 4, 2019 · 101 comments

Comments

Projects
None yet
6 participants
@whedon
Copy link
Collaborator

commented Feb 4, 2019

Submitting author: @1313e (Ellert van der Velden)
Repository: https://github.com/1313e/PRISM
Version: v1.0.0
Editor: @arokem
Reviewer: @fonnesbeck
Archive: 10.5281/zenodo.2572736

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf"><img src="http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf/status.svg)](http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@fonnesbeck, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

Please try and complete your review in the next two weeks

Review checklist for @fonnesbeck

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v1.0.0)?
  • Authorship: Has the submitting author (@1313e) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 4, 2019

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @fonnesbeck it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 4, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 4, 2019

@1313e

This comment has been minimized.

Copy link

commented Feb 4, 2019

As mentioned in the blog here (http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing),
it must be clear to all parties involved that a JOSS submission is associated with an AAS Journals submission.
Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@fonnesbeck @arokem Do you need the AAS Journals submission reference for this by any chance?

@arfon

This comment has been minimized.

Copy link
Member

commented Feb 5, 2019

Given that there is currently (as far as I know) not a defined way of doing so, I would like to mention here that there is an AAS submission associated with this JOSS submission.

@1313e many thanks for letting us know about this - I think you're the first!

@fonnesbeck - when you get a chance, please review this blog post from late last year: http://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing . In particular, please let us know if you have any objection to JOSS receiving a small donation from AAS publishing for this.

@1313e

This comment has been minimized.

Copy link

commented Feb 5, 2019

@1313e many thanks for letting us know about this - I think you're the first!

Yeah, the news about that collaboration came just in time before my paper submission, so I am not surprised I am the first one.

@arfon arfon added the AAS label Feb 5, 2019

@1313e

This comment has been minimized.

Copy link

commented Feb 7, 2019

@arfon May I btw mention that AAS Journals has no system in place at all that a submission is being accompanied by a JOSS submission?
I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

@arfon

This comment has been minimized.

Copy link
Member

commented Feb 7, 2019

I simply mentioned it in the extra notes to the editor, but I have not heard anything back yet.

Yes, I think this is still a work in progress.

/ cc @crawfordsm who is the software editor for AAS so he has some visibility into this review too.

@crawfordsm

This comment has been minimized.

Copy link

commented Feb 7, 2019

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

@1313e

This comment has been minimized.

Copy link

commented Feb 7, 2019

Apologies, unfortunately, I did not see a note about this on the AAS side. I'll have to follow up about that, but thanks for letting me know! As this is the first paper, it will be great to use as an example of the process.

@crawfordsm Do you, by any chance, need the submission reference, to make it easier to find which paper it actually is that I submitted to AAS Journals?

@crawfordsm

This comment has been minimized.

Copy link

commented Feb 7, 2019

It's okay, I have the information. I am the Scientific Editor for the paper, so I can monitor it here as well as on the AAS side. I realize now that I had just overlooked the information. As of right now, we do not have a formal process for this, but appreciate your patience as we figure out the details.

@arokem

This comment has been minimized.

Copy link

commented Feb 10, 2019

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

@arfon

This comment has been minimized.

Copy link
Member

commented Feb 10, 2019

Hi @arfon: thanks for the heads up about this. Do I understand correctly that the process here is essentially the same as for other JOSS papers? That is, review and acceptance here are not tied in any way to the AAS paper. Is that correct?

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

@arokem

This comment has been minimized.

Copy link

commented Feb 10, 2019

Understood! OK - @fonnesbeck - if this all sounds agreeable to you, please do go ahead with your review. Once the review process here is over, we can circle back and see what we need to do. Thanks!

@1313e

This comment has been minimized.

Copy link

commented Feb 11, 2019

That's mostly correct. We may want to hold this paper back from final acceptance until we know what the DOI for the AAS paper will be (so we can cite it).

@arfon That would be required anyways, given that I am citing the AAS Journals submission for the results.
I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

I hope that is okay btw, as I couldn't find any guidelines on this.

@arfon

This comment has been minimized.

Copy link
Member

commented Feb 11, 2019

I felt that putting them into the JOSS paper would be a bit pointless, given that they are already in the AAS submission and it would never fit into 1k words.

👍 yes that's right. We also don't allow scientific results to be published in JOSS :-)

I hope that is okay btw, as I couldn't find any guidelines on this.

I think you've got this right. @crawfordsm and I are in the process of drafting some guidelines that should assist future authors.

@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 15, 2019

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8
@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 15, 2019

I'm having problems running the example model. See 1313e/PRISM#7

@1313e

This comment has been minimized.

Copy link

commented Feb 16, 2019

@1313e the test suite fails on both Linux and macOS for me. pytest does not like the arguments:

pytest: error: unrecognized arguments: --mpl --pep8

@fonnesbeck You will have to install all requirements in requirements_dev.txt before running the pytests.
I could put that in the README for clarity, if required.

@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 16, 2019

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

@1313e

This comment has been minimized.

Copy link

commented Feb 17, 2019

Yes, at the moment the install instructions do not mention requirements_dev. Thanks.

@fonnesbeck Alright, I have added it to the README.
I have also "fixed" 1313e/PRISM#7.

@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 17, 2019

Tests now run after installing the dev requirements, but the tests fail.

@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 17, 2019

The submission satisfies all of the evaluation criteria. Once the issue above is addressed, I'm happy to recommend ACCEPT.

@1313e

This comment has been minimized.

Copy link

commented Feb 18, 2019

@fonnesbeck I have solved the problem with 1313e/PRISM#8, and also improved the README and docs on the topic of running the pytests.

@arokem Do you have an idea how I can fix the formatting error that whedon gives when compiling the paper.md?
It formats my last name incorrectly in the footer of every page (I already noticed that immediately after submission, but had not asked about it yet).

@arfon

This comment has been minimized.

Copy link
Member

commented Feb 18, 2019

@1313e - I’ll have to fix the name issue manually at the end (the automated proofs will continue to have this problem)

@fonnesbeck

This comment has been minimized.

Copy link

commented Feb 18, 2019

I am now able to run the test suite without issue. There are no other outstanding problems, so I recommend ACCEPT. cc @arfon

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 8, 2019

@1313e - am I correct that the AAS DOI for this paper will be 10.3847/1538-4365/ab1f7d?

If so, please add the following fields to the top of your paper.md file:

aas-doi: 10.3847/1538-4365/ab1f7d
aas-journal: Astrophysical Journal
@1313e

This comment has been minimized.

Copy link

commented Jun 9, 2019

@arfon Sure. However, shouldn't it be "Astrophysical Journal Supplement Series"?

@1313e

This comment has been minimized.

Copy link

commented Jun 9, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 9, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 9, 2019

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 9, 2019

Yes, please change the line: aas-journal: Astrophysical Journal to aas-journal: Astrophysics Journal Supplement Series

@1313e

This comment has been minimized.

Copy link

commented Jun 10, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 10, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 10, 2019

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 10, 2019

@1313e - I'm going to go ahead and accept this paper now as I believe the AAS paper is very close to being published too.

Apologies for the back and forth on this submission - you're the first author to follow the dual-publishing workflow in AAS/JOSS and we've been finding (and fixing) a few challenges along the way. Your patience has been much appreciated!

@arfon arfon added the accepted label Jun 10, 2019

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 10, 2019

@fonnesbeck - many thanks for your review here and to @arokem for editing this submission

@1313e - your paper is now accepted into JOSS ⚡️🚀💥

@arfon arfon closed this Jun 10, 2019

@whedon

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 10, 2019

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01229/status.svg)](https://doi.org/10.21105/joss.01229)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01229">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01229/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01229/status.svg
   :target: https://doi.org/10.21105/joss.01229

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

@1313e

This comment has been minimized.

Copy link

commented Jun 10, 2019

@arfon Will you also make sure that my last name is properly written in the published version?

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 10, 2019

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

@1313e

This comment has been minimized.

Copy link

commented Jun 11, 2019

@arfon Will you also make sure that my last name is properly written in the published version?

Certainly. Can you verify that it looks correct in the pdf here? http://joss.theoj.org/papers/f31acc7b1be30757526442034888aabf

Looks fine to me.

@1313e

This comment has been minimized.

Copy link

commented Jun 16, 2019

@arfon I was actually just looking at the paper, and noticed that the citations are not displayed properly. The first in every sequence is, but the other ones are not.
As far as I know, this was not the case with the automatically compiled versions.
Did I write the citations wrong?

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 16, 2019

@1313e - could you be more specific about what you're seeing that you believe is incorrect? i.e. give me an example of expected vs actual?

@1313e

This comment has been minimized.

Copy link

commented Jun 16, 2019

@arfon Well, the second sentence of the third paragraph shows up as "PRISM uses the Bayes linear approach (Goldstein & Wooff, 2007), emulation technique (Craig, Goldstein, Seheult, & Smith, 1996, p. @craig97) and history matching (Raftery, Givens, & Zeh, 1995, pp. @Craig96, @craig97)...".
I have the feeling that a few citations here were not actually formatted.

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 16, 2019

Ah, right. Sorry about that. The correct syntax for multiple citations is to separate them with semi-colons (e.g. [@Raftery95; @Craig96; @Craig97]). I've fixed up the paper locally in https://github.com/openjournals/joss-papers/blob/a66fbee618680adc4825d723edcbb40165e1dda0/joss.01229/10.21105.joss.01229.pdf and it should be correct now on the live site (note it can take a few hours for the updated PDF to show properly because of caching on our website).

@1313e

This comment has been minimized.

Copy link

commented Jun 16, 2019

@arfon Thanks for that, but I think you will have to manually set my last name in the footer again...
Similarly, I think that the BibTeX entry for the paper will also have my name incorrectly (as that is the case for the Hickle paper, published here in JOSS).
Is there anything on my part that I can do to avoid that?

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

@1313e

This comment has been minimized.

Copy link

commented Jun 17, 2019

The paper is now fixed.

Can you check that this looks OK? https://www.doi2bib.org/bib/10.21105/joss.01229

That link does not resolve.

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

Weird works fine for me. Can you try going to https://www.doi2bib.org/ and putting in 10.21105/joss.01229.

If that doesn't work, the output from the service (i.e. the bibtex entry from the Crossref metadata) is:

@article{vanderVelden2019,
  doi = {10.21105/joss.01229},
  url = {https://doi.org/10.21105/joss.01229},
  year = {2019},
  month = jun,
  publisher = {The Open Journal},
  volume = {4},
  number = {38},
  pages = {1229},
  author = {Ellert van der Velden},
  title = {Model dispersion with {PRISM}$\mathsemicolon$ an alternative to {MCMC} for rapid analysis of models},
  journal = {Journal of Open Source Software}
}

☝️does that look OK?

@1313e

This comment has been minimized.

Copy link

commented Jun 17, 2019

Uhm, my last name should be in braces, as otherwise it will not be formatted correctly.
Besides that, it looks good.

It does seem though that for my paper, my name is at least written correctly.
The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

@arfon

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

OK, thanks. I don't have any control over where https://www.doi2bib.org puts the braces but the metadata looks correct.

The JOSS paper for the Hickle package (21105/joss.01115) does not render it properly on that website, giving it as "Ellert Van Velden" again.

OK, that's fixed for 10.21105/joss.01115 in https://github.com/openjournals/joss-papers/blob/master/joss.01115/10.21105.joss.01115.crossref.xml#L43-L47 but it will take a few hours to update on https://www.doi2bib.org .

@1313e

This comment has been minimized.

Copy link

commented Jun 17, 2019

Alright, thanks.
I know from experience that having no braces can sometimes mess it up when using natbib for example, but I guess that if the metadata is correct, such a thing would not be hard to notice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.