[REVIEW]: DrWatson: the perfect sidekick for your scientific inquiries #2673
Comments
|
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @jpfairbanks, @kescobo it looks like you're currently assigned to review this paper 🎉. Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post. ⭐ Important ⭐ If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿 To fix this do the following two things:
For a list of things I can do to help you, just type: For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type: |
|
@whedon generate pdf |
|
PDF failed to compile for issue #2673 with the following error: sh: 0: getcwd() failed: No such file or directory |
|
|
👋 Hi @jpfairbanks and @kescobo. Thank you for agreeing to review this submission. Please work your way through the checklist above of items. Please feel free to raise technical issues on the repository, and procedural questions here. Thanks! |
|
Does JOSS have the concept of "corresponding author" and/or author order conventions? |
|
I would say no, that's up to the authors themselves. |
Related to [JOSS review](openjournals/joss-reviews#2673 (comment)) I've made a few minor edits to the manuscript. A couple of additional points: - I think it would also be good to mention in the `Comparison with existing software section`, where you discuss that many projects use multiple programming languages, that julia actually has really good interop with other languages. via eg PyCall and RCall. - In the same section, I think cwl is orthogonal to the purpose of DrWatson. The idea with CWL and other pipeline-type software (see [this discourse](https://discourse.julialang.org/t/rfc-mr-phelps-a-distributed-workflow-orchestrator/) thread for a bunch of other examples) is the run the same analysis pipeline on lots of different datasets. There's obviously some parallels with DrWatson in terms of reproducibility and project management, but they seem like overall they're targeting different concerns. - Prior to lines 134/135 where you have commands from the julia pkg REPL, it might be worth mentioning that, and providing a link to Pkg documentation. The other code in the manuscript is intended to be put inside julia scripts, and non-julia users are unlikely to be familiar with the `pkg>` prompt convention. - Similarly, the output in ln115-121 is not clearly distinct from the code starting at ln109. This may be a limitation of the document rendering engine, but it might be clearer if you write something like: ```` ```julia @TagSave(savename(parameters, "jld2"), Dict("data" => [1,2,3])); load(savename(parameters, "jld2")) # load back saved data ``` yielding the output: ``` Dict{Symbol,Any} with 6 entries: :gitcommit => "v1.13.0-1-g3a5364f" :script => "docs/build/string#3" :data => [1, 2, 3] :gitpatch => "" ``` ```` I have a few additional thoughts for features that would be useful, but in my mind they are not essential and the feature set currently available is very useful and more than sufficient for publication.
|
I've completed my review - pending the changes requested here, I'm happy to sign off. |
|
So I have a question about scope. Is the JOSS paper format intended to be a complete description of the code? How much of the "explaining how to use the software" should be delegated to the docs (which will stay up to date). |
|
Thanks @kescobo! @jpfairbanks: That's a great question. There should be one or a few code snippets giving a feeling for how to use the code, to explain the goals etc. of the software. The docs should have a complete API description etc. |
|
@jpfairbanks You can find review criteria here: |
|
The idea is for the paper to be relatively short, with the main documentation in the software repository itself. |
|
Thanks, I had read some of that language above in this thread and got confused because it talks about API docs, but I’m supposed to check that the API is documented in the actual manual. Not that the paper contains docs. This now makes sense. BTW I can’t mark the checkboxes in the checklist. Am I supposed to have different repo privs or something? |
|
OK, let me know if the wording can be improved. I believe that you should have got an invitation to the JOSS account giving you those privileges? |
* Paper review Related to [JOSS review](openjournals/joss-reviews#2673 (comment)) I've made a few minor edits to the manuscript. A couple of additional points: - I think it would also be good to mention in the `Comparison with existing software section`, where you discuss that many projects use multiple programming languages, that julia actually has really good interop with other languages. via eg PyCall and RCall. - In the same section, I think cwl is orthogonal to the purpose of DrWatson. The idea with CWL and other pipeline-type software (see [this discourse](https://discourse.julialang.org/t/rfc-mr-phelps-a-distributed-workflow-orchestrator/) thread for a bunch of other examples) is the run the same analysis pipeline on lots of different datasets. There's obviously some parallels with DrWatson in terms of reproducibility and project management, but they seem like overall they're targeting different concerns. - Prior to lines 134/135 where you have commands from the julia pkg REPL, it might be worth mentioning that, and providing a link to Pkg documentation. The other code in the manuscript is intended to be put inside julia scripts, and non-julia users are unlikely to be familiar with the `pkg>` prompt convention. - Similarly, the output in ln115-121 is not clearly distinct from the code starting at ln109. This may be a limitation of the document rendering engine, but it might be clearer if you write something like: ```` ```julia @TagSave(savename(parameters, "jld2"), Dict("data" => [1,2,3])); load(savename(parameters, "jld2")) # load back saved data ``` yielding the output: ``` Dict{Symbol,Any} with 6 entries: :gitcommit => "v1.13.0-1-g3a5364f" :script => "docs/build/string#3" :data => [1, 2, 3] :gitpatch => "" ``` ```` I have a few additional thoughts for features that would be useful, but in my mind they are not essential and the feature set currently available is very useful and more than sufficient for publication. * use Normal package instead of command line * attempt to separate two code blocks * Highlight Julia's interoperability Co-authored-by: Datseris <datseris.george@gmail.com>
For this paper you can consider myself the corresponding author (if need be). |
|
I think that this is a great tool and a great write-up that sufficiently introduces the software, situates it in the relevant prior art and motivates me to adopt the software based on claimed benefits. The benefits of better simulation data management is obvious to anyone who has analyzed the performance, accuracy, or predictions of a computer simulation. There is no empirical evidence of efficiency or utility presented, but I am not sure that publication of this material requires that kind of empirical validation and defer to the editor's judgement. The biggest issue from a methodological perspective is that Dr. Watson avoids the use of databases for managing the data collected during the experiments. The current practice for simulation data management is to create files with paths that encode information about the parameters and with contents that encode information about the results. Dr. Watson automates and formalizes these ad-hoc techniques. In that automating and formalizing ad hoc techniques can improve the reliability of the methods, this is good. But as a field, computer science has solved this data management problem with an entirely different technique. Relational Database Management Systems (RDBMS) are the standard technology designed for managing data when you need persistence and correctness with the ability to store data quickly and query against it later. RDBMSes support flexible query languages for answering complex, ad-hoc queries about the data and indexing to make it fast. Why does Dr. Watson not use a database for storing this data? The paper should address this approach and why it was not selected. The current approach for simulation data management is ad hoc file naming conventions and folder structures containing structured or semi-structured data in the output files. So any solution that starts by improving this solution is welcome. However for more fruitful analysis of the resulting data, a proper data ingestion with ETL into an SQL database would allow for more rich querying. This could lead to easier feature development down the road when it comes to experiment analysis. This concern could be addressed by identifying previous attempts to use RDBMSes in simulation data management and explaining why they failed, or discussing this path as a viable approach that has not yet been explored, and leaving integration with existing RDBMS techniques for future work, once the RDBMS ecosystem in Julia has matured. |
|
Thank you for the review @jpfairbanks , CaosDB is a RDBMS (I believe, although the creators name it Scientific Data Management System), and has been developed by scientists for scientists. We plan to integrate DrWatson with CaosDB.jl once a sufficiently stable Julia implementation exists. I believe once such an integration is in order, your concerns about the lack of data managing capabilities of DrWatson will be addressed. However, I do think this will take substantial amount of effort and thus it is unlikely that we are able to do it during the period of this review. |
|
That sounds great, then I would request that the article describe the limitation of DrWatson for organizing or querying the resulting data and that you would use DrWatson for tracking the data generation processes and then ETL it into CaosDB for querying. Describing that as future work would be appropriate. |
|
@whedon set https://zenodo.org/record/4133570 as archive |
|
https://zenodo.org/record/4133570 doesn't look like an archive DOI. |
|
@Datseris Could you please check that the version number is correct |
|
@whedon accept |
|
No archive DOI set. Exiting... |
|
@whedon set 10.5281/zenodo.4133570 as archive |
|
OK. 10.5281/zenodo.4133570 is the archive. |
|
@whedon accept |
|
|
|
👋 @openjournals/joss-eics, this paper is ready to be accepted and published. Check final proof 👉 openjournals/joss-papers#1875 If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1875, then you can now move forward with accepting the submission by compiling again with the flag |
|
@Datseris - could you please merge this PR which fixes some issues with your DOIs? JuliaDynamics/DrWatson.jl#208 |
|
Thank you @arfon , I just did! |
|
@whedon check references |
|
|
@whedon accept |
|
|
👋 @openjournals/joss-eics, this paper is ready to be accepted and published. Check final proof 👉 openjournals/joss-papers#1879 If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1879, then you can now move forward with accepting the submission by compiling again with the flag |
|
|
@whedon accept deposit=true |
|
|
🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦 |
|
🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨 Here's what you must now do:
Any issues? Notify your editorial technical team... |
|
@jpfairbanks, @kescobo, @apdavison - many thanks for your reviews here and to @dpsanders for editing this submission ✨ @Datseris - your paper is now accepted into JOSS ⚡🚀💥 |
|
🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉 If you would like to include a link to your paper from your README use the following code snippets: This is how it will look in your documentation: We need your help! Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:
|
|
Congrats @tamasgal and @Datseris! Many thanks to @kescobo, @jpfairbanks and @apdavison for your reviews. |
Submitting author: @tamasgal (Tamás Gál)
Repository: https://github.com/JuliaDynamics/DrWatson.jl
Version: 1.16.3
Editor: @dpsanders
Reviewers: @jpfairbanks, @kescobo, @apdavison
Archive: 10.5281/zenodo.4133570
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@jpfairbanks & @kescobo, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @dpsanders know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @jpfairbanks
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @kescobo
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @apdavison
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
The text was updated successfully, but these errors were encountered: