Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: Sensie: Probing the sensitivity of neural networks #2180

Closed
38 tasks done
whedon opened this issue Apr 30, 2020 · 79 comments
Closed
38 tasks done

[REVIEW]: Sensie: Probing the sensitivity of neural networks #2180

whedon opened this issue Apr 30, 2020 · 79 comments
Assignees
Labels
accepted published Papers published in JOSS recommend-accept Papers recommended for acceptance in JOSS. review

Comments

@whedon
Copy link

whedon commented Apr 30, 2020

Submitting author: @coljac (Colin Jacobs)
Repository: https://github.com/coljac/sensie
Version: v1.0.0
Editor: @gkthiruvathukal
Reviewers: @ejhigson, @omshinde
Archive: 10.5281/zenodo.3894083

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/cabd37234d4a4d19b9d047907bc81ebe"><img src="https://joss.theoj.org/papers/cabd37234d4a4d19b9d047907bc81ebe/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/cabd37234d4a4d19b9d047907bc81ebe/status.svg)](https://joss.theoj.org/papers/cabd37234d4a4d19b9d047907bc81ebe)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ejhigson, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.

Please try and complete your review in the next two weeks

Review checklist for @ejhigson

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@coljac) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @omshinde

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@coljac) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
@whedon
Copy link
Author

whedon commented Apr 30, 2020

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ejhigson it looks like you're currently assigned to review this paper 🎉.

Important

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

@whedon
Copy link
Author

whedon commented Apr 30, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK

MISSING DOIs

- https://doi.org/10.1007/978-3-319-44781-0_8 may be missing for title: Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers
- https://doi.org/10.1109/iccv.2017.74 may be missing for title: Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization

INVALID DOIs

- None

@whedon
Copy link
Author

whedon commented Apr 30, 2020

@arfon
Copy link
Member

arfon commented Apr 30, 2020

@ejhigson - thanks again for agreeing to review this submission for JOSS.

Please carry out your review in this issue by updating the checklist above and giving feedback in this issue. The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

Any questions/concerns please let myself or @gkthiruvathukal know.

@arfon
Copy link
Member

arfon commented Apr 30, 2020

👋 @nilesh-patil @sssomani @chekoduadarsh @omshinde - would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The package being reviewed here is Sensie, a Python package for probing the sensitivity of neural networks.

@omshinde
Copy link

omshinde commented Apr 30, 2020

Hi @arfon ! Thanks for consideration. Yes, I would be interested to review this submission for JOSS. This would be my maiden review for JOSS and I foresee a great learning experience ahead. Please let me know of the further steps. Thanks!

@arfon
Copy link
Member

arfon commented Apr 30, 2020

Sounds good to me, thanks @omshinde. @gkthiruvathukal - do you want to take it from here? (You can do @whedon add @omshinde as reviewer) but then you'll have to edit the top of the issue to add a checklist for them.

@ejhigson
Copy link

ejhigson commented May 3, 2020

Hi @coljac - congratulations on a really neat piece of software. I definitely see the usefulness of this package!

I took an initial look over it and have a few suggestions:

  1. For the installation, I suggest adding your requirements to setup.py using the install_requires property so they are installed automatically (in addition to having them in the requirements.txt). You also may want to list your package on PyPI so users can install it with pip without doing a git clone.

  2. Forgive me if I have missed them but I can't see any instructions for how to run your automated tests in the docs? When I try with nosetests I get some errors but maybe I am doing something wrong?

  3. I would suggest adding a bit more detail to your community guidelines at https://sensie.readthedocs.io/en/latest/index.html. The review criteria is "Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support". Some of this is covered in your "Contact" section but maybe you can add more.

  4. In the paper, you survey the literature on testing DNN sensitivity but you don't mention if there are any other software packages for doing this similar to sense. Please could you add this? If the answer is no then its worth saying so explicitly.

  5. One of the references in the paper, "Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers", is missing a DOI.

@coljac
Copy link

coljac commented May 5, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented May 5, 2020

@coljac
Copy link

coljac commented May 5, 2020

Hi @ejhigson,

Thanks for taking the time to review Sensie, it's much appreciated. I have responded to your initial feedback as follows:

For the installation, I suggest adding your requirements to setup.py using the install_requires property so they are installed automatically (in addition to having them in the requirements.txt). You also may want to list your package on PyPI so users can install it with pip without doing a git clone.

Thanks for this suggestion. I have updated setup.py accordingly, and adjusted the documentation to make the installation procedure clearer.

Forgive me if I have missed them but I can't see any instructions for how to run your automated tests in the docs? When I try with nosetests I get some errors but maybe I am doing something wrong?

pip install pytest and then pytest test from the repo root should run the tests successfully.

I have also indicated this in the documentation:

"Optionally, install pytest with pip install pytest, then run the tests with pytest test from the repository root."

I would suggest adding a bit more detail to your community guidelines at https://sensie.readthedocs.io/en/latest/index.html. The review criteria is "Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support". Some of this is covered in your "Contact" section but maybe you can add more.

I have updated the documentation to read as follows on readthedocs and GitHub:

"Issues, Questions and Contributions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Any problems or questions? Email colin@coljac.net, or open an issue on GitHub at https://github.com/coljac/sensie.

Contributions are welcome and encouraged. Fork the GitHub repository to your own machine, make some changes, and push your work back up to the fork and open a pull request so that I can review the changes."

In the paper, you survey the literature on testing DNN sensitivity but you don't mention if there are any other software packages for doing this similar to sense. Please could you add this? If the answer is no then its worth saying so explicitly.

I'm not aware of anything that does what Sensie does precisely. I added the following to the paper:

"Several quality software packages exist to probe ANN internals, such as Innvestigate, tf-explain, and keras-vis. These packages are geared towards traditional image-based applications, and cannot explore relationships between the neural network performance and arbitrary properties of the examples presented to the network. Sensie is designed to fill that gap."

One of the references in the paper, "Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers", is missing a DOI.

Fixed.

@coljac
Copy link

coljac commented May 5, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented May 5, 2020

PDF failed to compile for issue #2180 with the following error:

Error reading bibliography ./paper.bib (line 63, column 3):
unexpected "f"
expecting space, ",", white space or "}"
Error running filter pandoc-citeproc:
Filter returned error status 1
Looks like we failed to compile the PDF

@coljac
Copy link

coljac commented May 5, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented May 5, 2020

@ejhigson
Copy link

ejhigson commented May 7, 2020

Hi @coljac - thank you very much for these changes!

The install and tests work perfectly for me, and the community guidelines additions look good

"Several quality software packages exist to probe ANN internals, such as Innvestigate, tf-explain, and keras-vis. These packages are geared towards traditional image-based applications, and cannot explore relationships between the neural network performance and arbitrary properties of the examples presented to the network. Sensie is designed to fill that gap."

Thank you for adding this to the paper - I think it is very useful background information.

Rereading the paper I spotted a couple of minor typos: you are missing a comma after Innvestigate and in the caption of Figure 1 I think "ann" should be "an". Aside from this I think the paper looks great and is very well written and clear. I particularly like the examples in the paper.

Regarding documentation: this looks mostly complete but there are a few methods missing documentation as can be seen at https://sensie.readthedocs.io/en/latest/sensie.html and in the source code. Please could you check it over and fill in the missing methods' documentation?

@ejhigson
Copy link

ejhigson commented May 7, 2020

Also, looking at the examples on the jupyter notebooks, I think "MNIST example" is missing an
import numpy as np
statement in the first cell? With this addition it runs perfectly for me. "CIFAR10 example" also works perfectly for me. However "VGG example" seems to need some files from your local drive at /home/coljac/Downloads/imagenet/gorillas(?) Is this example notebook meant to be runnable by other people?

@gkthiruvathukal
Copy link

gkthiruvathukal commented May 7, 2020

@arfon Just letting you know, with this comment, that I am resuming editorial activities and will see this and other submissions through the process. It's good to see @ejhigson has been offering good feedback for @coljac, who is working to address the issues.

@arfon
Copy link
Member

arfon commented May 7, 2020

@arfon Just letting you know, with this comment, that I am resuming editorial activities and will see this and other submissions through the process.

👍thanks @gkthiruvathukal! One quick reminder (and possibly stating the obvious here) - we still need a second reviewer for this one.

@gkthiruvathukal
Copy link

gkthiruvathukal commented May 7, 2020

@arfon Is it easy to add a reviewer after review has started? I seem to recall that wasn't easy to do in the past. I'll try to find a second. Let's hope more people are becoming available as some of the COVID-19 measures begin to lift.

@coljac
Copy link

coljac commented May 8, 2020

Hi @ejhigson,

Thanks again for the follow up comments - and the bug/typo finding.

  • I added the missing numpy import in the notebook example.
  • I fixed the typos indicated.

Regarding the VGG example, for copyright reasons I didn't check in/make available the collection of gorilla pictures I used for this example, and so I've left that as an exercise for the reader (though they can of course see how things work from the state of the notebook checked in). I've added a note to this effect to the top. Let me know if you think this is an acceptable solution; otherwise, I can hunt down some images (in a different class) whose copyright I can vouch for.

As for the documentation, I ensured there are docstrings for all the methods and they show up on readthedocs; I also put in docstrings for all the underscore-methods for those accessing the source. That should all be in order now.

Thanks again for the comments and the attention to detail.

@ejhigson
Copy link

ejhigson commented May 8, 2020

@coljac thank you very much for your latest changes - they all look good to me! I think the VGG example now works well with the note at the top you just added telling people they can use their own images instead.

I am satisfied that you have met all the JOSS criteria - the paper is well written and the performance all seems as expected. Congratulations on a very neat software package and a nice paper!

@arfon
Copy link
Member

arfon commented May 8, 2020

@arfon Is it easy to add a reviewer after review has started? I seem to recall that wasn't easy to do in the past. I'll try to find a second. Let's hope more people are becoming available as some of the COVID-19 measures begin to lift.

You can do @whedon add xxx as reviewer here which will update the assignments but unfortunately you'll need to manually create a checklist for the second reviewer by editing the issue body at the top of this thread.

@coljac
Copy link

coljac commented May 8, 2020

@ejhigson Thanks for the nice comments and the work as reviewer, it's much appreciated!

@omshinde
Copy link

omshinde commented May 8, 2020

Hi @arfon and @gkthiruvathukal , sorry for the long communication gap from my side. I couldn't start the review as I did not receive any reviewer invitation as mentioned above. Also, I am not sure whether I would require permissions to edit the review checklist or I can make the checklist in a comment. Please let me know if I should still review the work and make the checklist here itself in the comment.

Thanks.
Thanks to @coljac and @ejhigson for the contributions. :)

@coljac
Copy link

coljac commented May 18, 2020

Sorry to chime in here; I noticed @omshinde was willing to review but was waiting for confirmation. Does anything further need to be done @gkthiruvathukal?

@whedon
Copy link
Author

whedon commented Jun 18, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1007/978-3-319-44781-0_8 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK

MISSING DOIs

- https://doi.org/10.1109/iccv.2017.74 may be missing for title: Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization

INVALID DOIs

- None

@Kevin-Mattheus-Moerman
Copy link
Member

Kevin-Mattheus-Moerman commented Jun 18, 2020

@coljac some editorial comments on the paper:

  • Check/fix double occurrence of directly in: Some well-known methods for visualising and interpreting the outputs of DNNs include directly inspecting the learned features of a model directly
  • Please define the acronym ANN (or should it be DNN?)
  • If the software you cite in footnotes has papers to cite please do add them.
  • The paper is about to be processed for acceptance in JOSS. Please proofread the paper yourself again. In particular check the author names, affiliations, and acknowledgement section.

Once you've made these changes please call @whedon generate pdf to update the paper (use @whedon check references if you add new citations).

@coljac
Copy link

coljac commented Jun 18, 2020

@Kevin-Mattheus-Moerman

Done and done. Thanks for catching a couple of issues.

@coljac
Copy link

coljac commented Jun 18, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented Jun 18, 2020

@coljac
Copy link

coljac commented Jun 18, 2020

@whedon check references

@whedon
Copy link
Author

whedon commented Jun 18, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1007/978-3-319-44781-0_8 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK

MISSING DOIs

- https://doi.org/10.1109/iccv.2017.74 may be missing for title: Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization

INVALID DOIs

- None

@Kevin-Mattheus-Moerman
Copy link
Member

Kevin-Mattheus-Moerman commented Jun 19, 2020

@coljac great I ticked all those boxes. Can you check that last potentially missing DOI? Is it this paper: https://ieeexplore.ieee.org/document/8237336 ?

@coljac
Copy link

coljac commented Jun 19, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented Jun 19, 2020

@coljac
Copy link

coljac commented Jun 19, 2020

@whedon check references

@whedon
Copy link
Author

whedon commented Jun 19, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1007/978-3-319-44781-0_8 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK
- 10.1109/ICCV.2017.74 is OK

MISSING DOIs

- None

INVALID DOIs

- None

@coljac
Copy link

coljac commented Jun 19, 2020

@Kevin-Mattheus-Moerman I fixed the erroneous DOI before realising I removed the reference from the body of the paper. I've updated accordingly.

@coljac
Copy link

coljac commented Jun 19, 2020

@whedon generate pdf

@whedon
Copy link
Author

whedon commented Jun 19, 2020

@coljac
Copy link

coljac commented Jun 19, 2020

@whedon check references

@whedon
Copy link
Author

whedon commented Jun 19, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1007/978-3-319-44781-0_8 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK

MISSING DOIs

- None

INVALID DOIs

- None

@Kevin-Mattheus-Moerman
Copy link
Member

Kevin-Mattheus-Moerman commented Jun 19, 2020

@whedon accept

@whedon
Copy link
Author

whedon commented Jun 19, 2020

Attempting dry run of processing paper acceptance...

@whedon whedon added the recommend-accept Papers recommended for acceptance in JOSS. label Jun 19, 2020
@whedon
Copy link
Author

whedon commented Jun 19, 2020

Reference check summary:

OK DOIs

- 10.1016/j.dsp.2017.10.011 is OK
- 10.1109/ICIP.2016.7532763 is OK
- 10.1371/journal.pone.0130140 is OK
- 10.1007/978-3-319-44781-0_8 is OK
- 10.1002/widm.1349 is OK
- 10.3847/1538-4365/ab26b6 is OK
- 10.1093/mnras/stz272 is OK
- 10.1093/mnras/stx1492 is OK

MISSING DOIs

- None

INVALID DOIs

- None

@whedon
Copy link
Author

whedon commented Jun 19, 2020

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1500

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1500, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

@Kevin-Mattheus-Moerman
Copy link
Member

Kevin-Mattheus-Moerman commented Jun 19, 2020

@whedon accept deposit=true

@whedon
Copy link
Author

whedon commented Jun 19, 2020

Doing it live! Attempting automated processing of paper acceptance...

@whedon whedon added accepted published Papers published in JOSS labels Jun 19, 2020
@whedon
Copy link
Author

whedon commented Jun 19, 2020

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

@whedon
Copy link
Author

whedon commented Jun 19, 2020

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

  1. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02180 joss-papers#1501
  2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02180
  3. If everything looks good, then close this review issue.
  4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

@whedon
Copy link
Author

whedon commented Jun 19, 2020

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02180/status.svg)](https://doi.org/10.21105/joss.02180)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02180">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02180/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02180/status.svg
   :target: https://doi.org/10.21105/joss.02180

This is how it will look in your documentation:

DOI

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted published Papers published in JOSS recommend-accept Papers recommended for acceptance in JOSS. review
Projects
None yet
Development

No branches or pull requests

7 participants