New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: GAMA: Genetic Automated Machine learning Assistant #1132

Closed
whedon opened this Issue Dec 17, 2018 · 48 comments

Comments

Projects
None yet
6 participants
@whedon
Copy link
Collaborator

whedon commented Dec 17, 2018

Submitting author: @PGijsbers (Pieter Gijsbers)
Repository: https://github.com/PGijsbers/GAMA
Version: v0.1.0
Editor: @arokem
Reviewer: @jsgalan
Archive: 10.5281/zenodo.2545472

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71"><img src="http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71/status.svg)](http://joss.theoj.org/papers/74cfa7e5796270cd2a96005370973c71)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jsgalan, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @arokem know.

Please try and complete your review in the next two weeks

Review checklist for @jsgalan

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v0.1.0)?
  • Authorship: Has the submitting author (@PGijsbers) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Dec 17, 2018

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @jsgalan it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Dec 17, 2018

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Dec 17, 2018

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Dec 24, 2018

Just making sure I did not miss anything, but there is not something I should be doing right now, correct?

@arfon

This comment has been minimized.

Copy link
Member

arfon commented Dec 24, 2018

Just making sure I did not miss anything, but there is not something I should be doing right now, correct?

Correct, we're waiting on @jsgalan to complete their review (by updating the checklist above).

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 14, 2019

@jsgalan
I am sorry to bother and I know there have been holidays in-between, but given the indicated review period I felt compelled to ask. I was wondering if there is any estimate of when you plan to work on this? Or should we see if we can find a substitute?
edit: to be clear, there are no bad feelings here, just looking for an update.

@jsgalan

This comment has been minimized.

Copy link
Collaborator

jsgalan commented Jan 14, 2019

Hi all, i proceeded with the first steps of the review (installation and upgrades here:
installation-gama.txt )

Everything is installed correctly, however I get the following errors,

For Example #1
screen shot 2019-01-14 at 11 18 47 am

For Example #2
screen shot 2019-01-14 at 11 23 10 am

For Example #3
screen shot 2019-01-14 at 11 30 50 am

Any clues?

Best

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 15, 2019

That's odd, I can not recreate the issue locally (tried a clean install on a ubuntu docker image). It looks like you installed it to a 'gama' virtual environment, are you also running the scripts from that environment?

@jsgalan

This comment has been minimized.

Copy link
Collaborator

jsgalan commented Jan 15, 2019

Hi, i restarted the computer and everything worked fine :)

Here are my results for the tests provided in the documentation:

screen shot 2019-01-15 at 8 50 33 am

Had a few warnings for run-test2.txt and run-test5.txt

And the rest looked fine also for
run-test3.txt and example 4
screen shot 2019-01-15 at 11 15 32 am

@jsgalan

This comment has been minimized.

Copy link
Collaborator

jsgalan commented Jan 15, 2019

I have a few suggestions to be made to the authors:

1- add a few examples (not just the minimal example, but all five examples) to the Github repository. Also please check that the examples is wrongly edited, missing the last part of the statement

screen shot 2019-01-15 at 11 31 09 am

2- Provide a more detailed examples and explanations of the following sections: Using ARFF files, Logging, Log Visualization and GAMA Search Space Configuration.

3- The authors could add in the manuscript a few lines describing in detai

l logging and log visualization aspects, which can be interesting for a user/reader.

4- Documentation should be complete, in the comparison with other AutoML tools a paragraph is missing.
screen shot 2019-01-15 at 11 28 41 am

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 15, 2019

Good to hear it works now! Thank you for the feedback, I will ping you here when I am done resolving the issues. I expect to be able to finish them all by tomorrow.

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 16, 2019

Hi @jsgalan!
I processed your feedback best I could, but I still have some questions:

  1. What more detailed explanation are you missing for the log visualization?
  2. We mentioned the log visualization and type of questions you can currently answer with it in our paper submission. What in particular were you missing or expected to be changed?

I changed log levels of some statements so that anything of at least logging.INFO level will not end with a stream of warning statements. The reported crashes in the statements were harmless to the optimization process, but I concede that it is not a nice user experience :)
For all other aspects, I updated the docs.

Thanks again for your time.

@jsgalan

This comment has been minimized.

Copy link
Collaborator

jsgalan commented Jan 16, 2019

Hi all,

Sorry for being so vague on my descriptions. I will reformulate:

1- add a few examples (not just the minimal example, but all five examples) to the Github repository.

1.1-Also please check that the examples is wrongly edited, missing the last part of the statement <-- this was done! [check!]

2- What more detailed explanation are you missing for the log visualization? <- can this link or this information be found somewhere on the initial Github webpage

3- We mentioned the log visualization and type of questions you can currently answer with it in our paper submission. What in particular were you missing or expected to be changed?

The article currently reads:

In addition to its general use AutoML functionality, GAMA aims to serve AutoML researches as well. During the optimization process, GAMA keeps an extensive log of
progress made. Using this log, insight can be obtained on the behaviour of the population of pipelines. It can answer questions such as which mutation operator is most
effective, how fitness changes over time, and how much time each algorithm takes

I think is important to include in those lines regarding the aspects such as:

a) the figures/plots it produces. (a.e Fitness over number of evaluations vs No. evaluations, Pipeline length over number of evaluations vs No. evaluations and more importantly how Fitness over number of evaluations by main learner vs No. evaluations )

b) all the methods/models that can be learned (a.e NB, Decision Tree,Boosting, Random Forest, KNN, SVC, LogReg). This is not stated clearly in the documentation.

Sorry for being stubborn but I think this should be main focus of the article and be very well described in the Github website, showing all the full capabilities of the software and all the models that can be tested using the implementation.

After revising I think some extra homework...

4- There were .csv files generated that are not well described anywhere

screen shot 2019-01-16 at 1 35 00 pm

4.1- There were folders empty folders created with the same name of the .csv files.
4.2- Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

Hope this helps.

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 16, 2019

add a few examples (not just the minimal example, but all five examples) to the Github repository.

Could you elaborate what you mean specifically? I made three of them into separate scripts in the example folder and linked them from the README.md. What other examples are you referring to? Setting log output and adding an event hook?

can this link or this information be found somewhere on the initial Github webpage

It is referred to in the documentation but I will add it to the README.md, good call.

4, 4.1

I turned it off for examples now, as I don't think examples should produce extra files. I will add documentation regarding the files should you wish to produce them.

Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

Yes, using the 'cache_dir' hyperparameter when initializing a GamaClassifier or GamaRegressor object. I will make sure it also shows up in the regular documentation (not just API docs).

I'll make the changes tomorrow.

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 16, 2019

Sorry for being stubborn [...]

Also, no worries :) it's your job 👍 I appreciate the effort you put in

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 17, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 17, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 17, 2019

@whedon generate pdf

Sorry for the spam. It seems that Whedon did not take the last version last time? Missing a line-break.

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 17, 2019

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 17, 2019

Hi @jsgalan,

add a few examples (not just the minimal example, but all five examples) to the Github repository.

No news since last comment.

What more detailed explanation are you missing for the log visualization? <- can this link or this information be found somewhere on the initial Github webpage

Added link and example image to the README.md.

I think is important to include in those lines regarding the aspects such as:
a) the figures/plots it produces [..]
b) all the methods/models that can be learned [..]

I added a plot example to the paper and also explicitly named some algorithms as well as emphasize using scikit-learn algorithms (see the latest article proof).
Furthermore I also explicitly wrote out and linked some of the algorithms used in the documentation.
However, I think including all the different plots produced and algorithms used is not a great idea for three reasons:

  1. There are many different plots and algorithms, taking up a lot of space while I think the current selection gives a good impression.
  2. Precise plots and algorithms are not what defines the package. You are free to extract more information from logs and/or add your own algorithms or remove used ones.
  3. Algorithms and visualization are susceptible to change. Other machine learning packages might be incorporated and/or better visualizations might be developed.

I find the adjustments I made are a reasonable balance between providing too little information and all information. Please let me know if you disagree.

  1. There were .csv files generated that are not well described anywhere
    4.1 There were folders empty folders created with the same name of the .csv files.
    4.2- Can the user specify the folder/csv files names instead of automatically assigning the date of the experimental run?

They now have their section in the documentation.
After reconsideration, the .csv is no longer be generated as gama.log supersedes it (and is thus not documented).

@jsgalan

This comment has been minimized.

Copy link
Collaborator

jsgalan commented Jan 17, 2019

Hi all @arfon @PGijsbers

I just revised and all the changes suggested were made in documents and pdf, respectively. I am happy to say that from my part all the requirements were fulfilled.

However, I think including all the different plots produced and algorithms used is not a great idea for three reasons:
There are many different plots and algorithms, taking up a lot of space while I think the current selection gives a good impression.
Precise plots and algorithms are not what defines the package. You are free to extract more information from logs and/or add your own algorithms or remove used ones.
Algorithms and visualization are susceptible to change. Other machine learning packages might be incorporated and/or better visualizations might be developed

Answering to this comment, after my initial thoughts and replies, I imagined that the package can be (and will be extended) to include different types of tests or can be put into a greater pipeline using various packages, so i concur with your reasoning.

Happy to revise and test this novel tool that will be beneficial for the ML community.

Best

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 17, 2019

Thanks again for all your effort @jsgalan!

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 18, 2019

I just noticed @arokem was not pinged, but he is the editor, I figure he needs to be pinged too/instead. So.. there :)

@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Jan 20, 2019

Great work everyone! Thanks for the review @jsgalan!

@PGijsbers : your paper is ready to be processed for acceptance.

Could you please create an archive of the current state of your software (e.g., by creating a tag/version of the software and uploading that to zenodo).

Once you have a DOI for the software, please post it here, so we can add the archive to the paper.

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 21, 2019

Done. Zenodo directs me here but it doesn't show. I assume it will just take a few minutes?

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 21, 2019

Seems like it is all fixed @arokem

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 24, 2019

@whedon set 10.5281/zenodo.2545472 as archive

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 24, 2019

I'm sorry @PGijsbers, I'm afraid I can't do that. That's something only editors are allowed to do.

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 24, 2019

Actually not sure what is preferred here:

  • specifically this version: 10.5281/zenodo.2545472
  • link to latest: 10.5281/zenodo.2545471
@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Jan 27, 2019

I think that this one is fine: 10.5281/zenodo.2545472

This is the version you generated after incorporating all of the reviewer comments?

Could you please edit the metadata of the Zenodo page (title and authors), so that it matches the paper?

Thanks!

@PGijsbers

This comment has been minimized.

Copy link

PGijsbers commented Jan 28, 2019

Done!

@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Jan 30, 2019

@whedon set 10.5281/zenodo.2545472 as archive

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

OK. 10.5281/zenodo.2545472 is the archive.

@arokem arokem added the accepted label Jan 30, 2019

@arokem

This comment has been minimized.

Copy link
Collaborator

arokem commented Jan 30, 2019

Congratulations! Your paper is now ready to be accepted.

Stand by for EIC or an Associate EIC to drop by and finalize this.

@danielskatz

This comment has been minimized.

Copy link
Collaborator

danielskatz commented Jan 30, 2019

Thanks for editing @arokem
Thanks for reviewing @jsgalan
And thanks for submitting @PGijsbers

@danielskatz

This comment has been minimized.

Copy link
Collaborator

danielskatz commented Jan 30, 2019

@whedon accept

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

Attempting dry run of processing paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019


OK DOIs

- http://doi.org/10.1007/s10994-018-5735-z is OK

MISSING DOIs

- https://doi.org/10.3389/fninf.2014.00014 may be missing for title: Scikit-learn: Machine Learning in Python

INVALID DOIs

- None
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

Check final proof 👉 openjournals/joss-papers#460

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#460, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
@danielskatz

This comment has been minimized.

Copy link
Collaborator

danielskatz commented Jan 30, 2019

@whedon accept deposit=true

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

Doing it live! Attempting automated processing of paper acceptance...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

  1. Check final PDF and Crossref metadata that was deposited 👉 openjournals/joss-papers#461
  2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01132
  3. If everything looks good, then close this review issue.
  4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

@whedon

This comment has been minimized.

Copy link
Collaborator Author

whedon commented Jan 30, 2019

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01132/status.svg)](https://doi.org/10.21105/joss.01132)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01132">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01132/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01132/status.svg
   :target: https://doi.org/10.21105/joss.01132

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment