[REVIEW]: Auto-eD: A visual learning tool for automatic differentiation

[whedon]

Submitting author: @lindseysbrown (Lindsey Brown)
Repository: https://github.com/lindseysbrown/Auto-eD
Branch with paper.md (empty if default branch): master
Version: v1.0.2
Editor: @labarba
Reviewer: @rasbt, @Atcold
Archive: 10.5281/zenodo.6800009
Paper kind: software

⚠️ JOSE reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d"><img src="https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d/status.svg)](https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@rasbt & @Atcold, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @labarba know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @rasbt

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v1.0.0)? [see external #27]
• Authorship: Has the submitting author (@lindseysbrown) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?) [see external #28]
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified? [see external #32]
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support [see external #34]

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)? [see external #35]

Review checklist for @Atcold

Conflict of interest

• As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v1.0.0)? #164
• Authorship: Has the submitting author (@lindseysbrown) made substantial contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation? (and documentation is sufficient?)
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• Installation instructions: Is there a clearly stated list of dependencies? (Ideally these should be handled with an automated package management solution.)
• Example usage: Do the authors include examples of how to use the software?
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state the need for this software and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @rasbt, @Atcold it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 1305

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1109/5.58337 may be a valid DOI for title: Backpropagation through time: what it does and how to do it

INVALID DOIs

- None

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.43 s (161.2 files/s, 104633.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JavaScript                      15           6243           5772          22772
HTML                            23            425              5           3486
Python                           5            310            444           1423
CSS                              6            375             83           1192
reStructuredText                13            392            573            710
Markdown                         3             24              0            117
Jupyter Notebook                 1              0            891             99
TeX                              1              3              0             36
DOS Batch                        1              8              1             27
make                             1              4              6             10
Bourne Shell                     1              0              0              3
-------------------------------------------------------------------------------
SUM:                            70           7784           7775          29875
-------------------------------------------------------------------------------


Statistical information for the repository '12b028f6f1a1c86180d2f1bd' was
gathered on 2022/02/25.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
David Sondak                    17         11196            965           19.61
Rachel Moon                     38         11962           2524           23.36
Xinyue(Cynthia) Wang             5           948              2            1.53
lindseysbrown                  115         25380           9031           55.49

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
David Sondak              10904           97.4          7.1               16.81
Rachel Moon               11544           96.5          0.3               16.48
lindseysbrown             14516           57.2         11.2               16.33

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[labarba]

@rasbt, @Atcold — Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed! 🚀

[Atcold]

Drafting here my review

It's really hard to understand what the repo / project offers.
There seem to be two components: a Python library and a web app.
The web app is hard to use without a tutorial of sorts. The documentation seems more like a lecture (hence, its content should go in the paper, instead), and only later there is a demo of how to use the web app, actually documenting it.

The README.md or the PDF paper (or both) should clearly state how the project is organised and how one should consume the material. Right now, I count 3 places where to read text (paper, developer manual, documentation) and 2 places where to use code (web app, Python library).

If the project is about providing a tool for understanding automatic differentiation, then the repo should display a tutorial as the main component. Or, even better, the web app should walk you through a teaching experience, but I understand this would be harder to put together.

[rasbt]

@labarba General question about opening issues during the review. Should we be using the "convert to Issue" function here and tagging the author in that issue?

Or should we be creating the issues in the project's GitHub repo? Just wondering what the best practices are for JOSE.

EDIT:
NVM, I think it is the latter. Not sure why I thought otherwise.

[rasbt]

@Atcold I have a similar experience.

Looking at the GitHub repository, I think most of the files (except ./docs) are the ones that are uploaded to Heroku for creating the website? Ideally, it should be more clear what we are looking at in this repo. This could be addressed by clarifying the code structure in the Readme file.

@Atcold One of us should probably create an issue for that. Since you commented first and said it better, do you want to go ahead with that?

The other issue is that it's not quite clear what we are reviewing here. Maybe @labarba could also help clarifying this?

• Should we be reviewing the whole automatic differentiation lecture or just the components that use the web app?
• Should be reviewing the Python & web app code at all, or should we just focus on the web app interface as the main "product"?

[Atcold]

Yeah, I can open the issue, @rasbt.
And I agree with your sentiment of not knowing what we're asked to review.

[labarba]

Hi folks! If you open any issues, be sure that you do so in the target repository, and mention this issue so we get a cross-link between the two. Thanks!

[labarba]

The author submitted this under the "software" article type, not the "learning module," so I suppose the author intends this application to be used as software to support teaching and learning in machine learning courses. If the author does not make clear how they intend their contribution to be re-used, that is useful input for improvement.

[lindseysbrown]

@labarba Thanks for the clarification. Our submission spanned both so we weren't sure how to submit.

The main contribution is the web app to visualize automatic differentiation that could be used to support any class in which that is taught. We have additionally provided a learning module that is complemented by the use of the web app as an example of how the app could be incorporated. For advanced users with programming experience, we have also provided the underlying software package which can be run independently of the web app.

We can certainly edit the README/paper to make it more clear that you can pick which of these tools are most appropriate based on your level of familiarity with automatic differentiation and with programming:

• Complete beginners should use the Read the Docs for an introduction to automatic differentiation with examples and exercises, complemented by the web app.
• People familiar with automatic differentiation who want a tool to support their understanding should use the web app.
• Advanced programmers who want to explore automatic differentiation in code may use the full software package.

[whedon]

👋 @Atcold, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @rasbt, please update us on how your review is going (this is an automated reminder).

[lindseysbrown]

@Atcold I think you said you were planning on opening an issue, but I didn't see it in the target repo. I have now added text to both the README and paper (second paragraph in Content) detailing the three different levels of interaction with the software we've provided.

@rasbt @labarba For your reference that this change has been made. (Still not completely clear on if this was more appropriately classified as software or learning module since we did both.)

[rasbt]

Thanks for making those updates @lindseysbrown !!

Regarding the scope of the review. I am still unsure what to do here and would appreciate your feedback and clarification @labarba . I recently read through the learning modules and had some thoughts, but I am not sure if this is appropriate/within the scope of this review. When I am looking through the reviewer checklist at the top of this thread, the review is basically just about making sure the software works (i.e., installs correctly and works as promised). This is very different from a regular journal review, and I am happy to stick to that, but I want to make sure that this is indeed what you want us to do?

[labarba]

👋 hi! When we started JOSE, we planned to have two submission types: software papers, and papers about learning modules (teaching content of some size bigger than a lecture and smaller than a course). When submitting, authors choose one article type, and according to that, our editorial system posts the correct review checklist. On a couple of occasions, we've had submissions that bridge the two article types, but we don't have an editorial process for these edge cases.

I request that you use the review checklist for the software (as that is the submission type chosen by the authors) but comment here on your assessment of the other relevant parts of this project.

[labarba]

Hi @rasbt, @Atcold 👋 – how is it going with your reviews? I see you've checked off a bunch of items on your checklists. Are we waiting for the authors to make some revisions? Give us a quick update when you can!

[rasbt]

Hey @labarba , I have yet another question about the review process 😅. When I reviewed papers for JOSS and JOSE in the past, as far as I remember, there have always been unit tests (and even a CI) as it is best practice for open source.

In the reviewer guidelines it says

Tests: Are there automated tests or manual steps described so that the function of the software can be verified?

So, it does sound like unit tests (or a CI setup) are not required. Technically, one could walk through the examples in (DeveloperDocumentation.ipynb) and check that everything is correct and then rely on jupyter notebook file-diffs to check whether the results can be reproduced on one's own machine. However, this would be quite a burden on the reviewer/user/developer, and in the spirit of open source best-practices, I am wondering if it is okay to suggest to the package maintainers to add unit tests to test the software and verify the functionality on one's own machine?

[labarba]

Our submission guidelines are not exhaustive, because they cannot be, and we do rely on community standards. We should make reference to those community standards when our review guidelines are insufficient.

In regards to testing, the key is that to be eligible for publication, the software needs to adhere to minimum standards that enable reusability: the code should be immediately useful to others. For JOSE, the software should be useful in teaching and learning settings. If the learners do not need to look at or modify the code, as the tool is used more like an app, any tests and documentation target potential contributors, who need to have a way to check correctness. This may exclude unit tests, but at least should have some ability to do system or regression testing. Relying on Jupyter-notebook diffs is surely inadequate for that?

[rasbt]

It's sometimes tricky to strike the balance between being a critical reviewer and trying to improve the submission, but not coming across as asking for unreasonable things.

Thanks for the response on that. I do think that in this case unit tests not only help me as a reviewer, but it could also help users who want to run it locally and use their package verify that everything works as intended in their environment. And, of course it makes contributing and updating this package in the future easier.

I think in this case it doesn't have to be a comprehensive new suite of unit tests hooked up to a CI. A reasonable balance between extra work and utility would be to convert the current examples in the developer doc into unit tests so that someone can run them from their computer and verify the actual results match the expected results.

[whedon]

OK. 10.5281/zenodo.6800009 is the archive.

[labarba]

@lindseysbrown — We request that authors edit the metadata of the Zenodo deposit so title and author list match the JOSE paper. It's just cleaner that way as readers see these as part of the "same scholarly object." Could you do that? (Note that you may not want Zenodo to do automatic updates of versions with each release. Note also that Zenodo pulls every committer into the author list.)

[lindseysbrown]

@labarba I've edited the title and author list on Zenodo.

[labarba]

@whedon recommend-accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.48550/arXiv.1907.07587 is OK
- 10.1109/5.58337 is OK

MISSING DOIs

- None

INVALID DOIs

- 10.5555/3122009.3242010 is INVALID

[whedon]

👋 @openjournals/jose-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/jose-papers#99

If the paper PDF and Crossref deposit XML look good in openjournals/jose-papers#99, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[labarba]

Hmm. It doesn't look like JMLR mints Does. Do you want to instead link to the URL?
url = {http://jmlr.org/papers/v18/17-468.html}

[lindseysbrown]

https://dl.acm.org/doi/abs/10.5555/3122009.3242010

This doi seems to link to the source? I can do whatever you prefer.

[labarba]

Interesting. The DOI itself does not resolve at http://doi.org/10.5555/3122009.3242010

EDIT: that is the link that will be automatically added in compiling the PDF, so I think it is best to add it as URL and not a DOI.

[lindseysbrown]

Ok. Should I create a new release with the doi changed to url???

[labarba]

No need. The release and archive are really targeting the package. The paper will be deposited by JOSE with its own DOI and will be archived too.

[lindseysbrown]

Ok, then I've committed the new bib file.

[labarba]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[labarba]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSE! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.jose.00161 jose-papers#100
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/jose.00161
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[labarba]

Congratulations, @lindseysbrown, your JOSE paper is published!

Huge thanks to our Reviewers: @rasbt, @Atcold — we couldn't do this without you 🙏

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://jose.theoj.org/papers/10.21105/jose.00161/status.svg)](https://doi.org/10.21105/jose.00161)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/jose.00161">
  <img src="https://jose.theoj.org/papers/10.21105/jose.00161/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://jose.theoj.org/papers/10.21105/jose.00161/status.svg
   :target: https://doi.org/10.21105/jose.00161

This is how it will look in your documentation:

We need your help!

Journal of Open Source Education is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://jose.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-jose

[rasbt]

Wow, awesome, congrats @lindseysbrown, @rachelmoonstar, and David Sondak! 🎉🎊🍾

[Atcold]

🥳🥳🥳

[arfon]

@labarba – the submitting author contacted me (on admin@theoj.org) to ask for an update to their paper (see this change: lindseysbrown/Auto-eD@7c1eef5). This should be possible with a simple @editorialbot reaccept but before I do this I wanted to get an explicit 👍 from you.

[Atcold]

The URL points to a broken page. Update not approved.

[lindseysbrown]

I am able to view the updated url: https://autoed.onrender.com
Note that render can be slow to spin up on the free tier.
(The update was from the heroku link which is broken.)

[Atcold]

When I tried it was not working. Now it works.