[REVIEW]: ∂P: A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

[whedon]

Submitting author: @MikeInnes (Michael Innes)
Repository: https://github.com/MikeInnes/zygote-paper
Branch with paper.md (empty if default branch):
Version:
Editor: @vchuravy
Reviewers: @tlienart, @jiahao
Archive: Pending

Status

Status badge code:

HTML: <a href="https://submissions.juliacon.org/papers/e028589fc895cb2e350586f0f6081eb6"><img src="https://submissions.juliacon.org/papers/e028589fc895cb2e350586f0f6081eb6/status.svg"></a>
Markdown: [![status](https://submissions.juliacon.org/papers/e028589fc895cb2e350586f0f6081eb6/status.svg)](https://submissions.juliacon.org/papers/e028589fc895cb2e350586f0f6081eb6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@tlienart & @jiahao, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/JuliaCon/proceedings-reviews/invitations

The reviewer guidelines are available here: https://proceedings.juliacon.org/guide/reviewers. Any questions/concerns please let @vchuravy know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @tlienart

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@MikeInnes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

Review checklist for @jiahao

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@MikeInnes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @tlienart, @jiahao it looks like you're currently assigned to review this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[tlienart]

• License missing -- [review] License missing MikeInnes/zygote-paper#13
• a few issues running neural sde: [review] Add a basic Readme and fix the *.toml files (?) MikeInnes/zygote-paper#14

running code:

• neural SDE

[jiahao]

Disclosing potential COI with one author (CR) - we are writing a manuscript for publication (not submitted yet) - otherwise no other COIs to declare. (The PNAS guidelines make no mention of intended future co-authorship).

[tlienart]

Ok, I wasn't really sure where to put my review, the whole Whedon is a bit confusing (or requires to open a ton of issues everywhere which seems a bit dumb). If I do need to open a ton of small issues, let me know and I'll just do it.

Review summary

Accept. The paper makes a strong and clear case for differentiable programming explaining both its potential and the reason why Julia may be particularly well suited to it. It presents clear use cases where one could strongly benefit from the ability to compute gradients across complex programs/simulators efficiently showing how this can be leveraged to unite "classical" scientific computing with "modern" techniques in machine learning and advance both fields.

Minor comments

I don't have big comments, maybe things that can be clarified / extended upon and would facilitate the reading.

Clarifications

• in introduction: the opposition between deep learning as data driven and scientific computing very technical numerically intensive, is a bit weird;
• (2) adjoint sensitivity; it'd be good to add a line of two or explanations here for those like me who wouldn't be familiar with this; maybe also clarify what's p and u and how it can be interpreted so that it becomes clearer that "the term ... is the primitive of backprop"
• last paragraph before (3) it'd be nice to add a line or two on what is not directly easy to get with del P // what are the limitations as of today and why (to clarify the "almost all") is it just merely a case of implementing all the atomic differentials or are there other limitations?
• table 1; it may be better to present this as a small plot and a simple line fit + explain that the slope is roughly 570ns etc
• last paragraph before (3.5) you seem to hint at composition of different ADs; an explanation would be helpful; how do you do that and why does it work? can Zygote just compose with ForwardDiff? or are we talking about different modes within Zygote?

Typos

• section (1.1), but will depend on --> $x$ , same a bit later at $x = 1.0$
• section (2) pagraph before (2.1) --> "can then be compile ... optimize ... " grammar; a bit further "in tact"
• paragraph below fig 2 .. again the $x$
• write [\bullet]_2 instead of []_2 to be more readable
• probably better to write 1 \equiv \partial z / \partial z but whatever
• in the model that executes on TPU there seems to be a stray backslash on the line grads = \Zygoteplain...
• in the QML part before the Heisenberg chain, there's expectation value, I guess this could be either expectation or expected value? or is it just me?

• in NDE for finance, in the equation for V(S, t) there shouldn't be $d\nu$ twice, also you say it's a conditional expectation? conditional on what? is this with respect to a specific measure?

Cosmetics

• fix horizontal overflow in \partial f before section 3
• too much vertical spaces around equations // inconsistency (e.g. in SDE the top space is much larger than the bottom one)
• I don't know if it's an artifact of what LaTeX compiler whedon uses but courier-like characters are not vectorised, e.g. URLs in the references or code in the article, it's pretty ugly and decreases readability. Probably just a matter of using a proper tex compiler.

Apart from these very minor comments, congrats, I enjoyed reading the paper

[vchuravy]

@MikeInnes have you address the review comments?

[vchuravy]

According to a conversation on Slack this paper has been published elsewhere.

@whedon withdraw

[ViralBShah]

Just for the record. Published in AAAI-MPLS 2020

http://ceur-ws.org/Vol-2587/article_8.pdf

cc @ChrisRackauckas

[vchuravy]

@whedon withdraw

[whedon]

Paper withdrawn.