New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REVIEW]: BetaML: The Beta Machine Learning Toolkit, a self-contained repository of Machine Learning algorithms in Julia #2849
Comments
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ablaom, @ppalmes it looks like you're currently assigned to review this paper 🎉. Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post. ⭐ Important ⭐ If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿 To fix this do the following two things:
For a list of things I can do to help you, just type:
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
|
|
Okay, here's an update of my review from the pre-review thread What the package providesThe package under review provides pure-julia implementations of two
Existing machine learning toolboxes in JuliaFor context, consider the following multi-paradigm ML
In addition to these are several excellent and mature packages Disclaimer: I am a designer/contributor to MLJ. According to the JOSS requirements, Value as pedagogical toolI don't think there are many rigorous machine learning courses or edit Okay, I see that I missed the link to the course notes, as The remaining argument for BetaML's pedagogical value rests on a
Let me respond to these:
In summary, while existing toolboxes might present a course instructor ConclusionsTo meet the requirements of JOSS, I think either: (i) The BetaML Frankly, a believe a greater service to the Julia open-source software Sundry commentsI didn't have too much trouble installing the package or running the
|
👋 @ppalmes, please update us on how your review is going. |
👋 @ablaom, please update us on how your review is going. |
I would consider my initial review finished. I have left unchecked, " Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines" as I would prefer the editor make this call, based on my comments. I would say "yes", but according to the guidelines there should be "obvious research application". If research includes education, then yes, definitely scholarly. The other minor installation item needs addressing from author; I make some suggestions. |
I'll start my review next week. |
Dear, I am pretty new to this open-format of reviewing papers. Please let me know if and when I am supposed to reply, in particular if I need to wait for the second reviewer and/or the editors, thank you :-) |
I would suggest you can reply to @ablaom review questions/comments because it can help also hasten the review process so that I can just focus on those issues not covered by both of your conversations if I still need more clarifications. |
Yes, @sylvaticus please respond existing feedback while we are waiting for additioinal feedback from @ppalmes. Thanks. |
Author's response to @ablaom review 1Above all, I would like to thanks the reviewer for having taken the time to provide the review and the useful suggestions he brings. I have implemented most of them, as they helped improving the software. My detailed response is below.
While "most" of the functionality is indeed already present, from the user point of view, they are not necessarily accessed in the same way and for some functionality, like missing imputation using GMM models, I am not aware of implementations in Julia. Also the kind of output is often different from current implementations. For example most classifiers in BetaML report the whole PMF of the various items rather than the mode. Together with the fact that the function
I am sorry for the misunderstanding here. I am not affiliated with that course. The course referenced uses Python to teach the algorithms, while I believe a Julia approach, when dealing with the internals of the algorithms (conversely to "just" using some API), is more appropriate, this is why I translated, and generalised, the code in Julia.
While tools for automatic sampling and cross-validation may be in scope with BetaML, I believe that the added value for pipeling in a language like Julia is not so strong like it is for other programming languages.
I rephrased the readme.md of the package, as the project evolved from being a mere "rewriting" of algorithms in Julia.
I do appreciate both the Reviewer comments and the MLJ as a mature, state-of-the art framework, I just believe that there is space for a different approach with different user cases.
As specified on a previous comment, the focus is on usability, whether this is important for didactic or applied research purposes.
I added an "Alternative packages" section that lists the most relevant and mature Julia packages in the topics covered by BetaML.
Being a much smaller package than MLJ, I believe the "Installation" and "Loading the module(s)" (for the first point) and "Usage" (for the second one) in the documentation do suffice.
I added a basic Random Forest example in the Readme.md so to provide the readers of an overview of different techniques to analyse the same dataset (iris).
I now load the data using a path relative to the package base path. In this way the script should load the correct data whichever is the current directory from which it is called by the user.
Thank you. I did indeed add a simple partition function to allow partition multiple matrices in one line, e.g.
Having a manifest means that I need to keep it updated and the user understand what it is. A related issue is to guarantee that notebooks are kept in sync with the code. I noticed that the reviewer use Literate.jl, I may consider it, as it helps keeping the examples under testing control.
I agree and changed the name to |
Response to author's response to my review@sylvaticus Thank you for your response and addressing some of my criticisms. @terrytangyuan The author has not addressed, to my satisfaction, a central objection, which can be rephrased as this: To show the software meets a research need, it needs to be demonstrated that the software is substantially easier to use than the substantially more powerful alternatives (in the demonstrated absence of some other pedagogical value). The author agrees that there are much more powerful alternatives. However, I maintain it is not substantially easier to use or to learn MLBeta, as I detail in my rebuttals 1-3 to assertions in the paper. This said, as an author of one of the alternatives, I naturally find my package easier to use than one with which I am less familiar. It is possible that @sylvaticus feels the same way about MLBeta for the much the same reason. Perhaps @ppalmes would care to comment specifically on this question (see italics above). To be clear, I think the software and paper are quality products. I also do not dismiss the possibility that users might prefer a future enhanced version of the MLBeta to existing alternatives. I am simply questioning whether MLBeta meets the stated requirements of JOSS at this stage of its development. |
👋 @ppalmes - I think this review could definitely benefit from your input here 🙏 |
My decision is Major Revision. The main contribution of the package is the reimplementation in pure Julia of the various algorithms in supervised and unsupervised learning for teaching purposes. I agree with @ablaom that in terms of usability, other existing toolkit are more straightforward and consistent to use. Among the things that I consider to be a major issue is the absence of pipeline API. All related packages mentioned support this API which is a big factor for usability. Here are my list of suggestions:
|
👋 all, I'm stepping in here to assist @terrytangyuan who is struggling to make time for JOSS editorial work at this time. Firstly, @ppalmes and @ablaom, many thanks for your careful and constructive reviews. There is some excellent feedback here for the @sylvaticus. I do need to address one aspect of this feedback however, best captured in this comment:
I agree this is important but it's not a strict requirement for a works to be published in JOSS. Primarily, the review criteria around (Substantial Scholarly Effort)[https://joss.readthedocs.io/en/latest/review_criteria.html#substantial-scholarly-effort] are designed to exclude very minor software contributions which we don't believe add meaningful value for potential users of the tooling. Based on @sylvaticus' responses in this thread I do not believe this work falls into that category. JOSS' primary mission is to provide a mechanism for authors doing software work to receive career credit for their work, and in borderline situations such as this, we defer to the author's need/ability to be cited for their work. As such, on this criterion of Substantial scholarly effort I am making an editorial decision to allow this submission to move forward. That said, there is still a reasonable amount of feedback (especially that most recently from @ppalmes) that it would be good to hear your response to @sylvaticus. Could you please respond here with your thoughts and potential plans to address? |
Yeah, I am ok to proceed with publication. My suggestions are to make the work more usable to the wider community. It is usable at the current form and I believe that the work will continue to improve. |
Yes, as you can see in the commit log, I am actually still implementing the modifications required by the reviewers.. I created an interface to my models for one of the toolbox cited (these interfface has been already being pushed but still needs to be included in a release of BetaML) and I am implementing a more detailed set of tutorials. |
⚡ thanks for the feedback @sylvaticus, looking forward to seeing the updates! |
@afron Thanks for this clarification! Your statements makes perfect sense to me and I am very happy to see this work acknowledged through publication. |
Dear editor and reviewers, I have updated the library and the paper to account for the reviewers' comments:
I am confident that the modifications introduced will help the users of the library and I thanks the reviewers for the time they spent in suggesting the improvements to the library and their guidance in implementing them. |
@whedon generate pdf |
@arfon what are the next steps now ? Should I create a software deposit on zenodo ? |
@whedon generate pdf |
@sylvaticus - could you please clean up all of the comments in your paper? I was trying to give the Also, please add more information to your affiliations - I'm not sure what many of them are. |
@whedon generate pdf Done it. I have removed the comments and specified the full affiliation names. I am very sorry for the 6 affiliations (it's crazy, I know..) but that's the way we have been asked to sign our papers :-/ :
|
@sylvaticus - I made a few minor changes to your paper here: sylvaticus/BetaML.jl#23 . Once you have merged this PR, could you make a new release of this software that includes the changes that have resulted from this review. Then, please make an archive of the software in Zenodo/figshare/other service and update this thread with the DOI of the archive? For the Zenodo/figshare archive, please make sure that:
I can then move forward with accepting the submission. |
Hello, I have created release v0.5.1 of the software which includes the text corrections of @arfon (thank you!) and I have deposited it on Zenodo: https://doi.org/10.5281/zenodo.4730205 |
@whedon set 10.5281/zenodo.4730205 as archive |
I'm sorry @sylvaticus, I'm afraid I can't do that. That's something only editors are allowed to do. |
@whedon set 10.5281/zenodo.4730205 as archive |
OK. 10.5281/zenodo.4730205 is the archive. |
@whedon accept |
|
👋 @openjournals/joss-eics, this paper is ready to be accepted and published. Check final proof 👉 openjournals/joss-papers#2267 If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2267, then you can now move forward with accepting the submission by compiling again with the flag
|
|
@whedon accept deposit=true |
|
🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦 |
🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨 Here's what you must now do:
Any issues? Notify your editorial technical team... |
@ablaom, @ppalmes - many thanks for your reviews here and to @terrytangyuan for editing this submission. JOSS relies upon the volunteer efforts of people like you and we simply wouldn't be able to this without you ✨ @sylvaticus - your paper is now accepted and published in JOSS ⚡🚀💥 |
🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉 If you would like to include a link to your paper from your README use the following code snippets:
This is how it will look in your documentation: We need your help! Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:
|
Thanks everyone for your precious time and useful suggestion. I ran the cloc util again and the lines of code went from 2450 when starting the review to over 5000 now, most of which incorporates reviewers ideas and suggestions. |
Submitting author: @sylvaticus (Antonello Lobianco)
Repository: https://github.com/sylvaticus/BetaML.jl
Version: v0.2.2
Editor: @terrytangyuan
Reviewer: @ablaom, @ppalmes
Archive: 10.5281/zenodo.4730205
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@ablaom & @ppalmes, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @terrytangyuan know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @ablaom
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @ppalmes
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
The text was updated successfully, but these errors were encountered: