New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: kima: Exoplanet detection in radial velocities #487

Closed
whedon opened this Issue Dec 6, 2017 · 49 comments

Comments

Projects
None yet
5 participants
@whedon
Collaborator

whedon commented Dec 6, 2017

Submitting author: @j-faria (João Faria)
Repository: https://github.com/j-faria/kima
Version: v1.0.0
Editor: @arfon
Reviewers: @nespinoza
Archive: 10.6084/m9.figshare.6615350

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b396d6f8c5566bb67844f05bda0cbc8a"><img src="https://joss.theoj.org/papers/b396d6f8c5566bb67844f05bda0cbc8a/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b396d6f8c5566bb67844f05bda0cbc8a/status.svg)](https://joss.theoj.org/papers/b396d6f8c5566bb67844f05bda0cbc8a)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@barentsen, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arfon know.

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v1.0.0)?
  • Authorship: Has the submitting author (@j-faria) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Dec 6, 2017

Collaborator

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @barentsen it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As as reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
Collaborator

whedon commented Dec 6, 2017

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @barentsen it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As as reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Dec 6, 2017

Collaborator
Attempting PDF compilation. Reticulating splines etc...
Collaborator

whedon commented Dec 6, 2017

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Dec 6, 2017

Collaborator
https://github.com/openjournals/joss-papers/blob/joss.00487/joss.00487/10.21105.joss.00487.pdf
Collaborator

whedon commented Dec 6, 2017

https://github.com/openjournals/joss-papers/blob/joss.00487/joss.00487/10.21105.joss.00487.pdf
@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Dec 18, 2017

Member

Friendly reminder to take a look at this when you get a chance @barentsen 😁

Member

arfon commented Dec 18, 2017

Friendly reminder to take a look at this when you get a chance @barentsen 😁

@barentsen

This comment has been minimized.

Show comment
Hide comment
@barentsen

barentsen Dec 20, 2017

Collaborator

I expect to carry out this review on Dec 23-24. Apologies for the delay!

Collaborator

barentsen commented Dec 20, 2017

I expect to carry out this review on Dec 23-24. Apologies for the delay!

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria commented Dec 21, 2017

Thank you, @barentsen!

@barentsen

This comment has been minimized.

Show comment
Hide comment
@barentsen

barentsen Dec 24, 2017

Collaborator

I am opening a few issues on the repository now because I happen to have spare time, but, needless to say: I do not expect a quick reply! @j-faria please enjoy your holidays and don't let my activity trigger you to work over the xmas-nye period, we all need time off to preserve mental health! =)

Collaborator

barentsen commented Dec 24, 2017

I am opening a few issues on the repository now because I happen to have spare time, but, needless to say: I do not expect a quick reply! @j-faria please enjoy your holidays and don't let my activity trigger you to work over the xmas-nye period, we all need time off to preserve mental health! =)

@barentsen

This comment has been minimized.

Show comment
Hide comment
@barentsen

barentsen Dec 25, 2017

Collaborator

I installed and ran kima and identified four key issues which I would like to see addressed before continuing my review:

  • The examples do not compile (j-faria/kima#18 & j-faria/kima#19).
  • There are no tests (be it manual or automated) which can be used to verify the core functionality (j-faria/kima#21).
  • The package does not follow the best practices in terms of code re-use/duplication; I encourage the authors to turn the scripts directory into a Python package (j-faria/kima#20).
  • The paper/docs do not explain in sufficient detail how the package compares to similar software tools (j-faria/kima#24).

I do think astronomy would benefit from having a high-quality open source software tool which contains the features and ideas provided by kima (e.g. nested sampling and GPs), so I strongly encourage the authors to address these issues. I am happy to answer further questions and mentor the improvements; i.e. I would love to see this package accepted eventually!

Collaborator

barentsen commented Dec 25, 2017

I installed and ran kima and identified four key issues which I would like to see addressed before continuing my review:

  • The examples do not compile (j-faria/kima#18 & j-faria/kima#19).
  • There are no tests (be it manual or automated) which can be used to verify the core functionality (j-faria/kima#21).
  • The package does not follow the best practices in terms of code re-use/duplication; I encourage the authors to turn the scripts directory into a Python package (j-faria/kima#20).
  • The paper/docs do not explain in sufficient detail how the package compares to similar software tools (j-faria/kima#24).

I do think astronomy would benefit from having a high-quality open source software tool which contains the features and ideas provided by kima (e.g. nested sampling and GPs), so I strongly encourage the authors to address these issues. I am happy to answer further questions and mentor the improvements; i.e. I would love to see this package accepted eventually!

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Dec 28, 2017

Hi @barentsen, thank you for your thorough and very helpful review of kima!
I agree these four points you identified are necessary and useful for the package so I'll make the changes and add the features in the next few days.

I sincerely appreciate your message about work during the holiday period. For a PhD student like me, such a balance is an important and often forgotten skill that needs mastering, so it's nice to see it highlighted.

j-faria commented Dec 28, 2017

Hi @barentsen, thank you for your thorough and very helpful review of kima!
I agree these four points you identified are necessary and useful for the package so I'll make the changes and add the features in the next few days.

I sincerely appreciate your message about work during the holiday period. For a PhD student like me, such a balance is an important and often forgotten skill that needs mastering, so it's nice to see it highlighted.

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jan 7, 2018

Hi @barentsen,
I think I have now addressed all the issues you raised in your first review.

  • The C++ code, together with the examples, is now compiled in Travis automatically (and the related issues were fixed)
  • I created the pykima package as suggested, which now makes available a kima-showresults script and can also be imported from a Python interpreter. I'm not sure if it makes sense to add this package to pypi, so that it is pip-installable, but would be happy to do so.
  • Added some automated tests to Travis. These test the Python package and also compile and run a new example to check the default priors.
  • Added some words to the wiki with advantages and disadvantages of kima when compared with similar packages. Also rephrased a bit the paper to highlight kima's features. But I don't think a full list of similar packages should be in the paper.

j-faria commented Jan 7, 2018

Hi @barentsen,
I think I have now addressed all the issues you raised in your first review.

  • The C++ code, together with the examples, is now compiled in Travis automatically (and the related issues were fixed)
  • I created the pykima package as suggested, which now makes available a kima-showresults script and can also be imported from a Python interpreter. I'm not sure if it makes sense to add this package to pypi, so that it is pip-installable, but would be happy to do so.
  • Added some automated tests to Travis. These test the Python package and also compile and run a new example to check the default priors.
  • Added some words to the wiki with advantages and disadvantages of kima when compared with similar packages. Also rephrased a bit the paper to highlight kima's features. But I don't think a full list of similar packages should be in the paper.
@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jan 7, 2018

@whedon generate pdf

j-faria commented Jan 7, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Jan 7, 2018

Collaborator
Attempting PDF compilation. Reticulating splines etc...
Collaborator

whedon commented Jan 7, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Jan 7, 2018

Collaborator
https://github.com/openjournals/joss-papers/blob/joss.00487/joss.00487/10.21105.joss.00487.pdf
Collaborator

whedon commented Jan 7, 2018

https://github.com/openjournals/joss-papers/blob/joss.00487/joss.00487/10.21105.joss.00487.pdf
@barentsen

This comment has been minimized.

Show comment
Hide comment
@barentsen

barentsen Jan 7, 2018

Collaborator

I hope to continue the review soon, but unfortunately I'll be tied up with the AAS meeting during all of next week.

Collaborator

barentsen commented Jan 7, 2018

I hope to continue the review soon, but unfortunately I'll be tied up with the AAS meeting during all of next week.

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jan 18, 2018

Hi @barentsen! Sorry to put pressure, but do you think you will have some time to continue the review in the next few days? I'll have my PhD defense in the end of next week, and it would be awesome to "announce" kima there for the first time :)

j-faria commented Jan 18, 2018

Hi @barentsen! Sorry to put pressure, but do you think you will have some time to continue the review in the next few days? I'll have my PhD defense in the end of next week, and it would be awesome to "announce" kima there for the first time :)

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Feb 5, 2018

Member

Friendly reminder to get to this sometime soon @barentsen :-)

Member

arfon commented Feb 5, 2018

Friendly reminder to get to this sometime soon @barentsen :-)

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Feb 15, 2018

Hi @barentsen, @arfon, any ETA for continuing this review? Thank you

j-faria commented Feb 15, 2018

Hi @barentsen, @arfon, any ETA for continuing this review? Thank you

@barentsen

This comment has been minimized.

Show comment
Hide comment
@barentsen

barentsen Feb 16, 2018

Collaborator

I apologize for the delay. I expect to have time to continue the review on Tuesday.

Collaborator

barentsen commented Feb 16, 2018

I apologize for the delay. I expect to have time to continue the review on Tuesday.

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Mar 11, 2018

Member

👋@barentsen - please take a look at this soon please :-)

Member

arfon commented Mar 11, 2018

👋@barentsen - please take a look at this soon please :-)

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Mar 22, 2018

Hi @arfon, what is the standard procedure to move on with this review? If @barentsen doesn't have time to continue the review should we look for another referee? This has been idle for a very long time now.

j-faria commented Mar 22, 2018

Hi @arfon, what is the standard procedure to move on with this review? If @barentsen doesn't have time to continue the review should we look for another referee? This has been idle for a very long time now.

@nespinoza

This comment has been minimized.

Show comment
Hide comment
@nespinoza

nespinoza Apr 7, 2018

Collaborator

Hi @arfon, would love to!

Collaborator

nespinoza commented Apr 7, 2018

Hi @arfon, would love to!

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Apr 7, 2018

Member

@nespinoza - fantastic, thanks!!!

Member

arfon commented Apr 7, 2018

@nespinoza - fantastic, thanks!!!

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Apr 7, 2018

Member

@whedon assign @nespinoza as reviewer

Member

arfon commented Apr 7, 2018

@whedon assign @nespinoza as reviewer

@whedon whedon assigned arfon and unassigned arfon Apr 7, 2018

@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Apr 7, 2018

Collaborator

OK, the reviewer is @nespinoza

Collaborator

whedon commented Apr 7, 2018

OK, the reviewer is @nespinoza

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Apr 7, 2018

Member

@nespinoza - please take a look at the instructions at the top of this issue - any problems, just let me know.

Member

arfon commented Apr 7, 2018

@nespinoza - please take a look at the instructions at the top of this issue - any problems, just let me know.

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Apr 20, 2018

Hi @nespinoza, thank you for accepting to review kima. Please feel free to comment on the repository and let me know if I can answer any questions that might come up.

j-faria commented Apr 20, 2018

Hi @nespinoza, thank you for accepting to review kima. Please feel free to comment on the repository and let me know if I can answer any questions that might come up.

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon May 1, 2018

Member

👋 @nespinoza - when do you think you might be able to complete this review by?

Member

arfon commented May 1, 2018

👋 @nespinoza - when do you think you might be able to complete this review by?

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria May 11, 2018

👋 hi @nespinoza, a friendly reminder to take a look at this, please.

j-faria commented May 11, 2018

👋 hi @nespinoza, a friendly reminder to take a look at this, please.

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon May 29, 2018

Member

I just emailed @nespinoza directly to remind them about this.

Member

arfon commented May 29, 2018

I just emailed @nespinoza directly to remind them about this.

@nespinoza

This comment has been minimized.

Show comment
Hide comment
@nespinoza

nespinoza May 31, 2018

Collaborator

Dear @j-faria,

First of all, I apologize for taking so long in submitting my review; a lot has happened in the past few months and finding time to throughly go through the code and documentation was very hard. However, after running the examples, going through the documentation and most of the code, I can now (at last!) write my review.

Let me start the review by saying congratulations on putting this software together! It is definitely work that is going to be very useful for many people working on the area of planet detection and characterization. It is also very timely given the advent of new radial-velocity follow-up machines.

As the authors can see, I have marked most of the review points above as I think they comply with the standards of a JOSS publication. I think the documentation is very good, and details quite nicely what the code does and how it does it. However, although the code runs smoothly, I found the examples to be not as thoroughly explained as I would like, thinking as a new user that wants to use the code for the things the code is supposed to do. In what follows, I will outline why I haven't marked the points above and what I would like to see in order to mark them as "done":

  1. Functionality and Performance. There are two important claims in the paper that I think any user would like to see in action. The first --- and easiest to showcase --- is that kima (a) fits Keplerian models to data, and the second is that (b) the algorithm used by kima, Diffusive Nested Sampling, is "extremely efficient" at sampling the kind of posteriors that appear in the problem of fitting multiple-planets to a radial-velocity dataset. Although both are implicitly showcased by the code --- after inspecting the posterior samples and comparing them to some expected results --- I think these should be explicitly showcased by the software to any user that is thinking on or is going to use it.
  • For (a) I believe one of the outputs of kima should be a set of posterior realizations of the model over the radial-velocity measurements, i.e., the same plot it is shown on the kima repository (https://github.com/j-faria/kima).
  • For point (b), I must say it is hard to actually confirm this is the case for this particular case. For the purposes of the software, perhaps it would suffice to find a real-world example (or any of the examples that are already within the documentation), try to fit the dataset with some available software (e.g., any of the ones you mention in Section 1 of kima's documentation), and show that kima actually is more (or as) efficient and in what way. Alternatively, explaining why or in what sense Diffusive Nested Sampling is more efficient for this particular problem than other current solutions and the respective citation would be enough as well.
  1. Example usage I really liked the examples within the package because they are either previously generated datasets from other authors or real-life problems. However, I'm inclined to think that they are not very useful by themselves to a new user wanting to use kima for the first time. In particular (a) it is not explained what exactly are the inputs to kima and how exactly they should be feeded to kima (although it is obvious after checking the code that the inputs are times and radial-velocities, this might not be obvious to any user; e.g., some users might want to use some activity index instead of time as the independant variable in the GP, or add a linear trend with some activity index and so on. Also, does any format for times and radial-velocity work? Comma separated? Tabs? Note that examples should be applications of these instructions, and not instructions on how to use the codes themselves) and (b) although there are examples for setting different priors, there is no step-by-step example on how to setup a problem from scratch.
  1. Community guidelines I think most of the points are not explicitly stated in the wiki. So please:
  • Provide clear guidelines for how to contribute to the software.
  • How to report issues or problems with the software.
  • Who should users look for support.

I have not marked the Version review point, as the repository still doesn't have a release (I suppose because it is awaiting for review).

Best,

Néstor

Collaborator

nespinoza commented May 31, 2018

Dear @j-faria,

First of all, I apologize for taking so long in submitting my review; a lot has happened in the past few months and finding time to throughly go through the code and documentation was very hard. However, after running the examples, going through the documentation and most of the code, I can now (at last!) write my review.

Let me start the review by saying congratulations on putting this software together! It is definitely work that is going to be very useful for many people working on the area of planet detection and characterization. It is also very timely given the advent of new radial-velocity follow-up machines.

As the authors can see, I have marked most of the review points above as I think they comply with the standards of a JOSS publication. I think the documentation is very good, and details quite nicely what the code does and how it does it. However, although the code runs smoothly, I found the examples to be not as thoroughly explained as I would like, thinking as a new user that wants to use the code for the things the code is supposed to do. In what follows, I will outline why I haven't marked the points above and what I would like to see in order to mark them as "done":

  1. Functionality and Performance. There are two important claims in the paper that I think any user would like to see in action. The first --- and easiest to showcase --- is that kima (a) fits Keplerian models to data, and the second is that (b) the algorithm used by kima, Diffusive Nested Sampling, is "extremely efficient" at sampling the kind of posteriors that appear in the problem of fitting multiple-planets to a radial-velocity dataset. Although both are implicitly showcased by the code --- after inspecting the posterior samples and comparing them to some expected results --- I think these should be explicitly showcased by the software to any user that is thinking on or is going to use it.
  • For (a) I believe one of the outputs of kima should be a set of posterior realizations of the model over the radial-velocity measurements, i.e., the same plot it is shown on the kima repository (https://github.com/j-faria/kima).
  • For point (b), I must say it is hard to actually confirm this is the case for this particular case. For the purposes of the software, perhaps it would suffice to find a real-world example (or any of the examples that are already within the documentation), try to fit the dataset with some available software (e.g., any of the ones you mention in Section 1 of kima's documentation), and show that kima actually is more (or as) efficient and in what way. Alternatively, explaining why or in what sense Diffusive Nested Sampling is more efficient for this particular problem than other current solutions and the respective citation would be enough as well.
  1. Example usage I really liked the examples within the package because they are either previously generated datasets from other authors or real-life problems. However, I'm inclined to think that they are not very useful by themselves to a new user wanting to use kima for the first time. In particular (a) it is not explained what exactly are the inputs to kima and how exactly they should be feeded to kima (although it is obvious after checking the code that the inputs are times and radial-velocities, this might not be obvious to any user; e.g., some users might want to use some activity index instead of time as the independant variable in the GP, or add a linear trend with some activity index and so on. Also, does any format for times and radial-velocity work? Comma separated? Tabs? Note that examples should be applications of these instructions, and not instructions on how to use the codes themselves) and (b) although there are examples for setting different priors, there is no step-by-step example on how to setup a problem from scratch.
  1. Community guidelines I think most of the points are not explicitly stated in the wiki. So please:
  • Provide clear guidelines for how to contribute to the software.
  • How to report issues or problems with the software.
  • Who should users look for support.

I have not marked the Version review point, as the repository still doesn't have a release (I suppose because it is awaiting for review).

Best,

Néstor

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jun 6, 2018

Hi @nespinoza, thank you very much for the thorough review, I really appreciate it.

I agree completely with the three main points you raised as still missing. In the last couple of days I started addressing point 3, by creating issue and pull request templates, adding a CONTRIBUTING.md to the repo and changing the documentation. I will try to unify more clearly all these guidelines so that it becomes easier for a new user to contribute to kima.

To address point 2, I was planning to create a tutorial / getting started guide as you suggested. This is (in some form) already included in the wiki but the points you identified are indeed missing.

Point 1 is harder to address completely. Actually, there is already the possibility to plot a set of posterior realizations of the model over the RV measurements. It is currently undocumented though, so it's as good as not being there. But I will solve that very soon.
Regarding performance it will be challenging to choose a dataset that showcases well enough the capabilities of DNest4. Another possibility is to change our claims in the paper: currently kima stands apart from other packages because it can calculate the evidence for a 0, 1, 2, ... planet model and because it can sample the posterior for the number of planets Np. These two features are unique (to my knowledge). So those claims are more easily justified.

I will work on a complete answer to the points raised in your review in the upcoming two weeks.

Thank you again!

j-faria commented Jun 6, 2018

Hi @nespinoza, thank you very much for the thorough review, I really appreciate it.

I agree completely with the three main points you raised as still missing. In the last couple of days I started addressing point 3, by creating issue and pull request templates, adding a CONTRIBUTING.md to the repo and changing the documentation. I will try to unify more clearly all these guidelines so that it becomes easier for a new user to contribute to kima.

To address point 2, I was planning to create a tutorial / getting started guide as you suggested. This is (in some form) already included in the wiki but the points you identified are indeed missing.

Point 1 is harder to address completely. Actually, there is already the possibility to plot a set of posterior realizations of the model over the RV measurements. It is currently undocumented though, so it's as good as not being there. But I will solve that very soon.
Regarding performance it will be challenging to choose a dataset that showcases well enough the capabilities of DNest4. Another possibility is to change our claims in the paper: currently kima stands apart from other packages because it can calculate the evidence for a 0, 1, 2, ... planet model and because it can sample the posterior for the number of planets Np. These two features are unique (to my knowledge). So those claims are more easily justified.

I will work on a complete answer to the points raised in your review in the upcoming two weeks.

Thank you again!

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jun 15, 2018

Hi @nespinoza,
I think I have now addressed all the issues raised in the review.

  • There is a new example with data of 51 Peg.
    To go through this example step-by-step I created a new getting started guide, which is included in the documentation.
  • The kima-showresults script was changed to be a bit more intuitive
    and the docs are now more clear, in my opinion.
  • The new kima-showresults rv argument, in particular, addresses the question of plotting posterior realizations of the model over the radial-velocity measurements
  • The documentation was also expanded and made clear in other parts.
    In particular, I included a description of the type of input data kima expects,
    and expanded the pages about how to contribute and how to get support.
  • There are new README files in each example
    which can, in the future, be expanded with more details about the results of the analysis.
  • I changed slightly the paper and added citations to justify some of the claims.

Hopefully these changes make the package ready for publication.
Allow me to once again acknowledge your contribution as referee, as I really think the changes brought up by your review improved kima substantially.

Regarding the Version and the github release, I think I'll need @arfon's help on this.
My idea was to release v1.0 once the submission is accepted.

Cheers,
João

j-faria commented Jun 15, 2018

Hi @nespinoza,
I think I have now addressed all the issues raised in the review.

  • There is a new example with data of 51 Peg.
    To go through this example step-by-step I created a new getting started guide, which is included in the documentation.
  • The kima-showresults script was changed to be a bit more intuitive
    and the docs are now more clear, in my opinion.
  • The new kima-showresults rv argument, in particular, addresses the question of plotting posterior realizations of the model over the radial-velocity measurements
  • The documentation was also expanded and made clear in other parts.
    In particular, I included a description of the type of input data kima expects,
    and expanded the pages about how to contribute and how to get support.
  • There are new README files in each example
    which can, in the future, be expanded with more details about the results of the analysis.
  • I changed slightly the paper and added citations to justify some of the claims.

Hopefully these changes make the package ready for publication.
Allow me to once again acknowledge your contribution as referee, as I really think the changes brought up by your review improved kima substantially.

Regarding the Version and the github release, I think I'll need @arfon's help on this.
My idea was to release v1.0 once the submission is accepted.

Cheers,
João

@nespinoza

This comment has been minimized.

Show comment
Hide comment
@nespinoza

nespinoza Jun 18, 2018

Collaborator

Dear @j-faria,

Thank you very much for considering the points made in the review. I have gone through all of them, including running the 51 Peg example, and everything runs smoothly!

I have no more comments to give; I think the software is in its final form in terms of a JOSS publication and, @arfon, @j-faria, I'm happy to recommend kima for publication once (1) the final PDF is generated, (2) the first version is released (I suppose we need @arfon to double-check this). @j-faria, as before, congratulations on coding kima; it will be very useful for many of us working on exoplanets, and is a tool that definitely covers huge needs in the area of exoplanet discovery and characterization!

Before finishing the review, one (optional, of course, as I have already recommended the software for publication) suggestion for @j-faria on kima: an explaination of each possible column that the posterior_sample.txt that kima generates for the posterior samples of the model would be great. I know this is obvious for most of us working on exoplanets (e.g., K is obviously the RV semi-amplitude, P the period, etc.), but it might not be for everyone!

All the best,
Néstor

Collaborator

nespinoza commented Jun 18, 2018

Dear @j-faria,

Thank you very much for considering the points made in the review. I have gone through all of them, including running the 51 Peg example, and everything runs smoothly!

I have no more comments to give; I think the software is in its final form in terms of a JOSS publication and, @arfon, @j-faria, I'm happy to recommend kima for publication once (1) the final PDF is generated, (2) the first version is released (I suppose we need @arfon to double-check this). @j-faria, as before, congratulations on coding kima; it will be very useful for many of us working on exoplanets, and is a tool that definitely covers huge needs in the area of exoplanet discovery and characterization!

Before finishing the review, one (optional, of course, as I have already recommended the software for publication) suggestion for @j-faria on kima: an explaination of each possible column that the posterior_sample.txt that kima generates for the posterior samples of the model would be great. I know this is obvious for most of us working on exoplanets (e.g., K is obviously the RV semi-amplitude, P the period, etc.), but it might not be for everyone!

All the best,
Néstor

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria commented Jun 18, 2018

@whedon generate pdf

@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Jun 18, 2018

Collaborator
Attempting PDF compilation. Reticulating splines etc...
Collaborator

whedon commented Jun 18, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon
Collaborator

whedon commented Jun 18, 2018

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jun 18, 2018

Thank you, @nespinoza!
The PDF is now generated (everything looks good I think). I will wait for instructions from @arfon on what to do next. And in the meantime I will add a description of the output files to the docs, I can probably do it before finishing the release.

j-faria commented Jun 18, 2018

Thank you, @nespinoza!
The PDF is now generated (everything looks good I think). I will wait for instructions from @arfon on what to do next. And in the meantime I will add a description of the output files to the docs, I can probably do it before finishing the release.

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Jun 18, 2018

Member

@j-faria - please make the new release and then create an archive of this release with Zenodo or figshare. Once you have done this, please let me know what the DOI of this archive is here.

Member

arfon commented Jun 18, 2018

@j-faria - please make the new release and then create an archive of this release with Zenodo or figshare. Once you have done this, please let me know what the DOI of this archive is here.

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jun 19, 2018

Hi @arfon, here's the DOI from figshare: 10.6084/m9.figshare.6615350

j-faria commented Jun 19, 2018

Hi @arfon, here's the DOI from figshare: 10.6084/m9.figshare.6615350

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Jun 19, 2018

Member

@whedon set 10.6084/m9.figshare.6615350 as archive

Member

arfon commented Jun 19, 2018

@whedon set 10.6084/m9.figshare.6615350 as archive

@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Jun 19, 2018

Collaborator

OK. 10.6084/m9.figshare.6615350 is the archive.

Collaborator

whedon commented Jun 19, 2018

OK. 10.6084/m9.figshare.6615350 is the archive.

@arfon arfon added the accepted label Jun 19, 2018

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Jun 19, 2018

Member

@nespinoza & @barentsen - many thanks for your reviews here

@j-faria - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00487 ⚡️ 🚀 💥

Member

arfon commented Jun 19, 2018

@nespinoza & @barentsen - many thanks for your reviews here

@j-faria - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00487 ⚡️ 🚀 💥

@arfon arfon closed this Jun 19, 2018

@whedon

This comment has been minimized.

Show comment
Hide comment
@whedon

whedon Jun 19, 2018

Collaborator

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippet:

[![DOI](http://joss.theoj.org/papers/10.21105/joss.00487/status.svg)](https://doi.org/10.21105/joss.00487)

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Collaborator

whedon commented Jun 19, 2018

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippet:

[![DOI](http://joss.theoj.org/papers/10.21105/joss.00487/status.svg)](https://doi.org/10.21105/joss.00487)

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

@j-faria

This comment has been minimized.

Show comment
Hide comment
@j-faria

j-faria Jun 21, 2018

Thank you @barentsen and @nespinoza for the reviews, they both increased the quality of kima substantially. Thank you @arfon also for your editorial effort and help.
(and thanks @whedon :)
I'm very happy to see this published!

j-faria commented Jun 21, 2018

Thank you @barentsen and @nespinoza for the reviews, they both increased the quality of kima substantially. Thank you @arfon also for your editorial effort and help.
(and thanks @whedon :)
I'm very happy to see this published!

@arfon

This comment has been minimized.

Show comment
Hide comment
@arfon

arfon Jun 21, 2018

Member

Sorry this took a while @j-faria - thanks for sticking with us!

Member

arfon commented Jun 21, 2018

Sorry this took a while @j-faria - thanks for sticking with us!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment