Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review #1 #7

Closed
distillpub-reviewers opened this Issue Sep 13, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@distillpub-reviewers
Copy link
Collaborator

distillpub-reviewers commented Sep 13, 2018

The following peer review was solicited as part of the Distill review process.

The reviewer chose to waive anonymity. Distill offers reviewers a choice between anonymous review and offering reviews under their name. Non-anonymous review allows reviewers to get credit for the service them offer to the community.

Distill is grateful to the reviewer, Abhinav Sharma, for taking the time to write such a thorough review.


The article is overall concise, readable and well written, but the organization has one small point of confusion. It starts out as a general method of qualitative analysis of different recurrent cells, but the latter part focuses on Nested LSTMs. The overall piece would be better served by either removing some of the specific details and focusing on the general method, or just focusing a single cell type in detail.

A small interaction improvement I would suggest (if the JS Distill permits allows it) is clicking on the “models should predict “learning” and is only given data until the first character.” link would both update the cursor and scroll the viewport to a position where all three examples are visible. At first, I simply didn’t notice that the diagram above was updated and was expecting a more visible update.

A small piece of visual design feedback regarding the diagram subtitled “Autocomplete” : you could lower the visual weight on the connectors. The connectors crossing over each other adds a lot of visual weight that doesn’t convey much information. Seeing all this crossing over made me wonder if the original sort on the blue “softmax bar” had inherent meaning. It looks like it doesn’t, and is specifically non-alphabetical because the nature of the autocomplete problem would put the 3 most likely outcomes too close to each other to discern. Another option instead of reordering outputs by probability is to preserve the order but do a visual “progress bar” or similar on the objects. Here’s a very crude illustration without the reordering (https://cl.ly/57e0d66dd0d6). Apologies for the lack of polish.

Overall this paper takes care with distilling a complex concept into a very intuitive visual formulation that should be very helpful. With some organizational improvements and addressing some minor issues, I think it would increase its impact.


Distill employs a reviewer worksheet as a help for reviewers.

The first three parts of this worksheet ask reviewers to rate a submission along certain dimensions on a scale from 1 to 5. While the scale meaning is consistently "higher is better", please read the explanations for our expectations for each score—we do not expect even exceptionally good papers to receive a perfect score in every category, and expect most papers to be around a 3 in most categories.

Any concerns or conflicts of interest that you are aware of?: No known conflicts of interest
What type of contributions does this article make?: Explanations of recurrent vanishing gradients, a new visualization technique to aid in qualitative comparisons of different approaches.

Advancing the Dialogue Score
How significant are these contributions? 4/5
Outstanding Communication Score
Article Structure 3/5
Writing Style 3/5
Diagram & Interface Style 3/5
Impact of diagrams / interfaces / tools for thought? 5/5
Readability 4/5
Scientific Correctness & Integrity Score
Are claims in the article well supported? 3/5
Does the article critically evaluate its limitations? How easily would a lay person understand them? 2/5
How easy would it be to replicate (or falsify) the results? 4/5
Does the article cite relevant work? 4/5
Does the article exhibit strong intellectual honesty and scientific hygiene? 4/5
@AndreasMadsen

This comment has been minimized.

Copy link
Member

AndreasMadsen commented Nov 9, 2018

It starts out as a general method of qualitative analysis of different recurrent cells, but the latter part focuses on Nested LSTMs.

I removed Nested LSTM almost completely from the conclusion, it should only serve as a recurrent unit for comparison since the article builds on the visualization done in the Nested LSTM paper.

A small interaction improvement I would suggest (if the JS Distill permits allows it) is clicking on the “models should predict “learning” and is only given data until the first character.” link would both update the cursor and scroll the viewport to a position where all three examples are visible. At first, I simply didn’t notice that the diagram above was updated and was expecting a more visible update.

I added that. I put a text in the caption. I changed the text into a clickable list, so the text and visualization can be seen at the same time on most screens.

A small piece of visual design feedback regarding the diagram subtitled “Autocomplete” : you could lower the visual weight on the connectors. The connectors crossing over each other adds a lot of visual weight that doesn’t convey much information. Seeing all this crossing over made me wonder if the original sort on the blue “softmax bar” had inherent meaning. It looks like it doesn’t, and is specifically non-alphabetical because the nature of the autocomplete problem would put the 3 most likely outcomes too close to each other to discern. Another option instead of reordering outputs by probability is to preserve the order but do a visual “progress bar” or similar on the objects. Here’s a very crude illustration without the reordering (https://cl.ly/57e0d66dd0d6). Apologies for the lack of polish.

I changed how the lines are drawn. I didn't add the “progress bar” visual.

Overall this paper takes care with distilling a complex concept into a very intuitive visual formulation that should be very helpful. With some organizational improvements and addressing some minor issues, I think it would increase its impact.

Thank you, for the nice words :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.