Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Review #1 #7
The following peer review was solicited as part of the Distill review process.
The reviewer chose to waive anonymity. Distill offers reviewers a choice between anonymous review and offering reviews under their name. Non-anonymous review allows reviewers to get credit for the service them offer to the community.
Distill is grateful to the reviewer, Abhinav Sharma, for taking the time to write such a thorough review.
The article is overall concise, readable and well written, but the organization has one small point of confusion. It starts out as a general method of qualitative analysis of different recurrent cells, but the latter part focuses on Nested LSTMs. The overall piece would be better served by either removing some of the specific details and focusing on the general method, or just focusing a single cell type in detail.
A small interaction improvement I would suggest (if the JS Distill permits allows it) is clicking on the “models should predict “learning” and is only given data until the first character.” link would both update the cursor and scroll the viewport to a position where all three examples are visible. At first, I simply didn’t notice that the diagram above was updated and was expecting a more visible update.
A small piece of visual design feedback regarding the diagram subtitled “Autocomplete” : you could lower the visual weight on the connectors. The connectors crossing over each other adds a lot of visual weight that doesn’t convey much information. Seeing all this crossing over made me wonder if the original sort on the blue “softmax bar” had inherent meaning. It looks like it doesn’t, and is specifically non-alphabetical because the nature of the autocomplete problem would put the 3 most likely outcomes too close to each other to discern. Another option instead of reordering outputs by probability is to preserve the order but do a visual “progress bar” or similar on the objects. Here’s a very crude illustration without the reordering (https://cl.ly/57e0d66dd0d6). Apologies for the lack of polish.
Overall this paper takes care with distilling a complex concept into a very intuitive visual formulation that should be very helpful. With some organizational improvements and addressing some minor issues, I think it would increase its impact.
Distill employs a reviewer worksheet as a help for reviewers.
The first three parts of this worksheet ask reviewers to rate a submission along certain dimensions on a scale from 1 to 5. While the scale meaning is consistently "higher is better", please read the explanations for our expectations for each score—we do not expect even exceptionally good papers to receive a perfect score in every category, and expect most papers to be around a 3 in most categories.
Any concerns or conflicts of interest that you are aware of?: No known conflicts of interest
I removed Nested LSTM almost completely from the conclusion, it should only serve as a recurrent unit for comparison since the article builds on the visualization done in the Nested LSTM paper.
I added that. I put a text in the caption. I changed the text into a clickable list, so the text and visualization can be seen at the same time on most screens.
I changed how the lines are drawn. I didn't add the “progress bar” visual.
Thank you, for the nice words :)