New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REVIEW]: matbench-genmetrics: A Python library for benchmarking crystal structure generative models using time-based splits of Materials Project structures #5618
Comments
Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks. For a list of things I can do to help you, just type:
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
|
|
Wordcount for |
|
Review checklist for @jamesrhesterConflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
|
Review checklist for @ml-evsConflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
|
@jamesrhester, @ml-evs, @mkhorton, thanks again for reviewing! Just checking in on your review. Please let me know if you have any questions about the process. Feel free to create issues in project repository directly or write them down as comments here, but please do link the issues in this review so it's easy to follow for everyone. @mkhorton, please go ahead and create your checklist first using the command |
Review checklist for @mkhortonConflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
|
👋 @jamesrhester , @mkhorton, please update us on how it's going with your reviews when you find the time |
Appreciate the reminder @phibeck, thank you -- on my radar, can't believe it's almost been a month already since agreeing to review! |
Likewise! |
Thank you all for getting the review started! As you work through your checklists, please feel free to comment and ask questions in this thread. You are encouraged to create issues in the repository directly. When you do, please mention Please let me know if you have any questions or if either of you requires some more time. |
Hi @phibeck, running through the checklist. Unfortunately I may have a conflict of interest: I have previously been on publications with the second author, Joseph Montoya. We worked together in the same research group until 2018 (i.e. outside the four year window in the COI policy), but the most recent paper we were on together actually only came out in 2021. |
If the work was done more then 4 years, but the paper appeared later, this is not a conflict for JOSS. The four years is about the collaborative relationship itself. |
Thank you, @danielskatz, for clarifying this question. Sounds like you don't have a COI here, @mkhorton. |
Just a heads-up (mostly for @phibeck) that I will restart my review on this, and will continue collecting small things in my old issue at sparks-baird/matbench-genmetrics#80 which wasn't previously linked here. |
@jamesrhester, @mkhorton & @ml-evs - could you provide an update on the progress of your review? Thank you! |
Getting onto this now. We've just had a big crystallography meeting that took up a lot of my cycles... |
As a crystallographer but non ML specialist, I found the "Statement of Need" lacked context. Line 22 in the paper made no sense to me, which is not good for the first sentence. I would therefore like one or two further sentences added at the beginning explaining how ML uses benchmarks, e.g "in ML, the result of a prediction is evaluated using benchmarks, which are then used to adjust the ML weights. Typically, a crystal structure ML model has used benchmarks from...." (which might show I have no idea what I'm talking about). I think this will better help readers to quickly determine whether or not the paper is relevant to them. Once this is done I'm ready to sign off on my review. |
Great, thanks for the update, @jamesrhester. @sgbaird, feel free to get started working on the comments and issues linked here by @jamesrhester and @ml-evs. Please update us here in this issue about the progress so we can keep track of the changes. |
@editorialbot check references |
|
@editorialbot generate pdf |
@sgbaird thanks! Here are a few more comments/suggestions for the manuscript. Please have a look when you have a moment.
Since I cannot find a record of the last two co-authors' contribution in your repository, could you please state their contributions here for the record of review? You can check out the guidance for authorship here: https://joss.readthedocs.io/en/latest/submitting.html#authorship Thanks! |
@editorialbot set v0.6.5 as version |
Done! version is now v0.6.5 |
@editorialbot set 10.5281/zenodo.10840604 as archive |
Done! archive is now 10.5281/zenodo.10840604 |
Done!
Agreed, removed
EDIT: Changed the wording to get the linebreak
Updated!
@JosephMontoya-TRI supplied code and an implementation related to the |
Had trouble getting the line break, so I updated the wording slightly to get it instead: sparks-baird/matbench-genmetrics@d05ead2 |
@editorialbot generate pdf |
Hi @sgbaird
Looks good, thanks!
It seems that the reference
Okay, thank you for clarifying! |
Sorry about that. Not sure what happened there. I added it in just now. Does it look ok? |
@editorialbot generate pdf |
No problem. Looks good now, thanks! |
@editorialbot check references |
|
@editorialbot recommend-accept |
|
|
👋 @openjournals/bcm-eics, this paper is ready to be accepted and published. Check final proof 👉📄 Download article If the paper PDF and the deposit XML files look good in openjournals/joss-papers#5342, then you can now move forward with accepting the submission by compiling again with the command |
I guess the Alverson reference needs to be updated to the published version. Will try to address shortly - no worries if too late. |
Submitting author: @sgbaird (Sterling Baird)
Repository: https://github.com/sparks-baird/matbench-genmetrics
Branch with paper.md (empty if default branch):
Version: v0.6.5
Editor: @phibeck
Reviewers: @ml-evs, @mkhorton, @jamesrhester
Archive: 10.5281/zenodo.10840604
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@ml-evs & @mkhorton & @jamesrhester, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @phibeck know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Checklists
📝 Checklist for @ml-evs
📝 Checklist for @jamesrhester
📝 Checklist for @mkhorton
The text was updated successfully, but these errors were encountered: