[REVIEW]: GAMA: Genetic Automated Machine learning Assistant

[whedon]

Submitting author: @PGijsbers (Pieter Gijsbers)
Repository: https://github.com/PGijsbers/GAMA
Version: v0.1.0
Editor: @arokem
Reviewer: @jsgalan
Archive: 10.5281/zenodo.2545472

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71"><img src="http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71/status.svg)](http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jsgalan, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @jsgalan

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v0.1.0)?
• Authorship: Has the submitting author (@PGijsbers) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @jsgalan it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[PGijsbers]

Just making sure I did not miss anything, but there is not something I should be doing right now, correct?

[arfon]

Just making sure I did not miss anything, but there is not something I should be doing right now, correct?

Correct, we're waiting on @jsgalan to complete their review (by updating the checklist above).

[PGijsbers]

@jsgalan
I am sorry to bother and I know there have been holidays in-between, but given the indicated review period I felt compelled to ask. I was wondering if there is any estimate of when you plan to work on this? Or should we see if we can find a substitute?
edit: to be clear, there are no bad feelings here, just looking for an update.

[jsgalan]

Hi all, i proceeded with the first steps of the review (installation and upgrades here:
installation-gama.txt )

Everything is installed correctly, however I get the following errors,

For Example #1

For Example #2

For Example #3

Any clues?

Best

[PGijsbers]

That's odd, I can not recreate the issue locally (tried a clean install on a ubuntu docker image). It looks like you installed it to a 'gama' virtual environment, are you also running the scripts from that environment?

[jsgalan]

Hi, i restarted the computer and everything worked fine :)

Here are my results for the tests provided in the documentation:

Had a few warnings for run-test2.txt and run-test5.txt

And the rest looked fine also for
run-test3.txt and example 4

[jsgalan]

I have a few suggestions to be made to the authors:

1- add a few examples (not just the minimal example, but all five examples) to the Github repository. Also please check that the examples is wrongly edited, missing the last part of the statement

2- Provide a more detailed examples and explanations of the following sections: Using ARFF files, Logging, Log Visualization and GAMA Search Space Configuration.

3- The authors could add in the manuscript a few lines describing in detai

l logging and log visualization aspects, which can be interesting for a user/reader.

4- Documentation should be complete, in the comparison with other AutoML tools a paragraph is missing.

[PGijsbers]

Good to hear it works now! Thank you for the feedback, I will ping you here when I am done resolving the issues. I expect to be able to finish them all by tomorrow.

[PGijsbers]

Hi @jsgalan!
I processed your feedback best I could, but I still have some questions:

1. What more detailed explanation are you missing for the log visualization?
2. We mentioned the log visualization and type of questions you can currently answer with it in our paper submission. What in particular were you missing or expected to be changed?

I changed log levels of some statements so that anything of at least logging.INFO level will not end with a stream of warning statements. The reported crashes in the statements were harmless to the optimization process, but I concede that it is not a nice user experience :)
For all other aspects, I updated the docs.

Thanks again for your time.

[jsgalan]

Hi all,

Sorry for being so vague on my descriptions. I will reformulate:

1- add a few examples (not just the minimal example, but all five examples) to the Github repository.

1.1-Also please check that the examples is wrongly edited, missing the last part of the statement <-- this was done! [check!]

2- What more detailed explanation are you missing for the log visualization? <- can this link or this information be found somewhere on the initial Github webpage

3- We mentioned the log visualization and type of questions you can currently answer with it in our paper submission. What in particular were you missing or expected to be changed?

The article currently reads:

In addition to its general use AutoML functionality, GAMA aims to serve AutoML researches as well. During the optimization process, GAMA keeps an extensive log of
progress made. Using this log, insight can be obtained on the behaviour of the population of pipelines. It can answer questions such as which mutation operator is most
effective, how fitness changes over time, and how much time each algorithm takes

I think is important to include in those lines regarding the aspects such as:

a) the figures/plots it produces. (a.e Fitness over number of evaluations vs No. evaluations, Pipeline length over number of evaluations vs No. evaluations and more importantly how Fitness over number of evaluations by main learner vs No. evaluations )

b) all the methods/models that can be learned (a.e NB, Decision Tree,Boosting, Random Forest, KNN, SVC, LogReg). This is not stated clearly in the documentation.

Sorry for being stubborn but I think this should be main focus of the article and be very well described in the Github website, showing all the full capabilities of the software and all the models that can be tested using the implementation.

After revising I think some extra homework...

4- There were .csv files generated that are not well described anywhere

4.1- There were folders empty folders created with the same name of the .csv files.
4.2- Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

Hope this helps.

[PGijsbers]

add a few examples (not just the minimal example, but all five examples) to the Github repository.

Could you elaborate what you mean specifically? I made three of them into separate scripts in the example folder and linked them from the README.md. What other examples are you referring to? Setting log output and adding an event hook?

can this link or this information be found somewhere on the initial Github webpage

It is referred to in the documentation but I will add it to the README.md, good call.

4, 4.1

I turned it off for examples now, as I don't think examples should produce extra files. I will add documentation regarding the files should you wish to produce them.

Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

Yes, using the 'cache_dir' hyperparameter when initializing a GamaClassifier or GamaRegressor object. I will make sure it also shows up in the regular documentation (not just API docs).

I'll make the changes tomorrow.

[PGijsbers]

Sorry for being stubborn [...]

Also, no worries :) it's your job 👍 I appreciate the effort you put in

[PGijsbers]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[PGijsbers]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[PGijsbers]

@whedon generate pdf

Sorry for the spam. It seems that Whedon did not take the last version last time? Missing a line-break.

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[PGijsbers]

Hi @jsgalan,

add a few examples (not just the minimal example, but all five examples) to the Github repository.

No news since last comment.

What more detailed explanation are you missing for the log visualization? <- can this link or this information be found somewhere on the initial Github webpage

Added link and example image to the README.md.

I think is important to include in those lines regarding the aspects such as:
a) the figures/plots it produces [..]
b) all the methods/models that can be learned [..]

I added a plot example to the paper and also explicitly named some algorithms as well as emphasize using scikit-learn algorithms (see the latest article proof).
Furthermore I also explicitly wrote out and linked some of the algorithms used in the documentation.
However, I think including all the different plots produced and algorithms used is not a great idea for three reasons:

1. There are many different plots and algorithms, taking up a lot of space while I think the current selection gives a good impression.
2. Precise plots and algorithms are not what defines the package. You are free to extract more information from logs and/or add your own algorithms or remove used ones.
3. Algorithms and visualization are susceptible to change. Other machine learning packages might be incorporated and/or better visualizations might be developed.

I find the adjustments I made are a reasonable balance between providing too little information and all information. Please let me know if you disagree.

4. There were .csv files generated that are not well described anywhere
   4.1 There were folders empty folders created with the same name of the .csv files.
   4.2- Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

They now have their section in the documentation.
After reconsideration, the .csv is no longer be generated as gama.log supersedes it (and is thus not documented).

[jsgalan]

Hi all @arfon @PGijsbers

I just revised and all the changes suggested were made in documents and pdf, respectively. I am happy to say that from my part all the requirements were fulfilled.

However, I think including all the different plots produced and algorithms used is not a great idea for three reasons:
There are many different plots and algorithms, taking up a lot of space while I think the current selection gives a good impression.
Precise plots and algorithms are not what defines the package. You are free to extract more information from logs and/or add your own algorithms or remove used ones.
Algorithms and visualization are susceptible to change. Other machine learning packages might be incorporated and/or better visualizations might be developed

Answering to this comment, after my initial thoughts and replies, I imagined that the package can be (and will be extended) to include different types of tests or can be put into a greater pipeline using various packages, so i concur with your reasoning.

Happy to revise and test this novel tool that will be beneficial for the ML community.

Best

[PGijsbers]

Thanks again for all your effort @jsgalan!

[PGijsbers]

I just noticed @arokem was not pinged, but he is the editor, I figure he needs to be pinged too/instead. So.. there :)

[arokem]

Great work everyone! Thanks for the review @jsgalan!

@PGijsbers : your paper is ready to be processed for acceptance.

Could you please create an archive of the current state of your software (e.g., by creating a tag/version of the software and uploading that to zenodo).

Once you have a DOI for the software, please post it here, so we can add the archive to the paper.

[PGijsbers]

Done. Zenodo directs me here but it doesn't show. I assume it will just take a few minutes?

[PGijsbers]

Seems like it is all fixed @arokem

[PGijsbers]

@whedon set 10.5281/zenodo.2545472 as archive

[whedon]

I'm sorry @PGijsbers, I'm afraid I can't do that. That's something only editors are allowed to do.

[PGijsbers]

Actually not sure what is preferred here:

• specifically this version: 10.5281/zenodo.2545472
• link to latest: 10.5281/zenodo.2545471

[arokem]

I think that this one is fine: 10.5281/zenodo.2545472

This is the version you generated after incorporating all of the reviewer comments?

Could you please edit the metadata of the Zenodo page (title and authors), so that it matches the paper?

Thanks!

[PGijsbers]

Done!

[arokem]

@whedon set 10.5281/zenodo.2545472 as archive

[whedon]

OK. 10.5281/zenodo.2545472 is the archive.

[arokem]

Congratulations! Your paper is now ready to be accepted.

Stand by for EIC or an Associate EIC to drop by and finalize this.

[danielskatz]

Thanks for editing @arokem
Thanks for reviewing @jsgalan
And thanks for submitting @PGijsbers

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- http://doi.org/10.1007/s10994-018-5735-z is OK

MISSING DOIs

- https://doi.org/10.3389/fninf.2014.00014 may be missing for title: Scikit-learn: Machine Learning in Python

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#460

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#460, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 openjournals/joss-papers#461
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01132
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01132/status.svg)](https://doi.org/10.21105/joss.01132)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01132">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01132/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01132/status.svg
   :target: https://doi.org/10.21105/joss.01132

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://www.flipcause.com/secure/cause_pdetails/Mjk3Nzk=