Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: Fashionable Modelling With Flux #56

Open
whedon opened this issue Oct 17, 2019 · 8 comments
Assignees
Labels

Comments

@whedon
Copy link
Collaborator

@whedon whedon commented Oct 17, 2019

Submitting author: @dhairyagandhi96 (Dhairya Gandhi)
Repository: https://github.com/dhairyagandhi96/flux-juliacon-paper.git
Editor: @vchuravy
Reviewer: @denizyuret
Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://submissions.juliacon.org/papers/626f91af151579cb19fcbf3e59586b3a"><img src="https://submissions.juliacon.org/papers/626f91af151579cb19fcbf3e59586b3a/status.svg"></a>
Markdown: [![status](https://submissions.juliacon.org/papers/626f91af151579cb19fcbf3e59586b3a/status.svg)](https://submissions.juliacon.org/papers/626f91af151579cb19fcbf3e59586b3a)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@denizyuret, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/JuliaCon/proceedings-reviews/invitations

The reviewer guidelines are available here: https://proceedings.juliacon.org/guide/reviewers. Any questions/concerns please let @vchuravy know.

Please try and complete your review in the next two weeks

Review checklist for @denizyuret

Conflict of interest

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Authorship: Has the submitting author (@dhairyagandhi96) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

  • Authors: Does the paper.tex file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
  • Page limit: Is the page limit for full papers respected by the submitted document?

Content

  • Context: is the scientific context motivating the work correctly presented?
  • Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
  • Results: are the results presented and compared to approaches with similar goals?
@whedon

This comment has been minimized.

Copy link
Collaborator Author

@whedon whedon commented Oct 17, 2019

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @denizyuret it looks like you're currently assigned to review this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
@whedon

This comment has been minimized.

Copy link
Collaborator Author

@whedon whedon commented Oct 17, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

@whedon whedon commented Oct 17, 2019

@denizyuret

This comment has been minimized.

Copy link
Collaborator

@denizyuret denizyuret commented Nov 1, 2019

Some questions / suggestions:

  • I am not sure what the page limit is. Assuming 8 pages is ok.
  • The repository url points to the paper repo, not Flux.jl. Assuming Flux.jl for other questions.
  • A lot of the checkmarks above are about the software, not the paper, I checked them assuming they are about Flux.jl.
  • The formatting seems off, lots of space at the bottom of each page, is this normal?
  • In listing 1 why (x/1) instead of just (x) is not clear.
  • In listing 3 @treelike ResidualBlock is not explained, and can be left out to avoid confusion.
  • Listing 3 may not be the best example to illustrate "By explicitly constraining types, the general Julia compiler can generate optimally packed memory representations."
  • Figure 1 can be placed closer to its example.
  • The variables in text (y_hat and y_dan_hat) do not match the ones in Listing 4.
  • Personally curious about AlphaGo Zero implementation: did you guys manage to have it converge for non-trivial games?
  • In general would be nice to have coverage/maturity/efficiency info in sections 3.X -- are these POC level implementations, are they mature/fast/complete/well-tested etc.? This info is missing with the exception of 3.5.
  • Listing 7 and Listing 8 are the same.
  • I am not sure if Listing 7 is a good motivator for auto-batching. ML data is typically vector even if we do not minibatch, and this function (like the previous leakyrelu example) would be applied with broadcasting.
  • Section 4: I have been thinking of the right way to do this: should we have the models in the Julia ecosystem at the package level (too many packages?), sub-package level (as done in Metalhead), split into multiple packages based on area (language, vision) etc. What should be the standard interface to train them, apply them, compose them, initialize them (possibly with pretrained weights) etc. Anyway this is a discussion worth having outside this paper...
  • Neither the paper nor Flux.jl has any pointers to Knet, is this intentional?
@dhairyagandhi96

This comment has been minimized.

Copy link

@dhairyagandhi96 dhairyagandhi96 commented Nov 5, 2019

Thanks for the review. Following this, we've made some changes to address some of the suggestions.

Re the AlphaGo point; we couldn't converge it for Go.

Per section 3; the GPU functionality, as well as the AD are mature. The acceleration with TPUs, outputting with JS and batching are working prototypes as of now. We plan to expand this functionality in the future.

With regards to Listing 7, it is not meant as a motivating example, rather an illustrative one. Broadcasting would do the same work internally, the vectorisation of the code is the larger question that we've tried to address in that section.

The missing cite to Knet is unintentional and we shall remedy that.

@whedon generate pdf

@vchuravy

This comment has been minimized.

Copy link

@vchuravy vchuravy commented Nov 11, 2019

@whedon generate pdf

@whedon

This comment has been minimized.

Copy link
Collaborator Author

@whedon whedon commented Nov 11, 2019

Attempting PDF compilation. Reticulating splines etc...
@whedon

This comment has been minimized.

Copy link
Collaborator Author

@whedon whedon commented Nov 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can’t perform that action at this time.