Review 3 - Anonymous

[colah]

The following peer review was solicited as part of the Distill review process. The review was formatted by the editor to help with readability.

The reviewer chose to keep anonymity. Distill offers reviewers a choice between anonymous review and offering reviews under their name. Non-anonymous review allows reviewers to get credit for the service them offer to the community.

Distill is grateful to the reviewer  for taking the time to write such a thorough review.

---

General Comments

Most of my concerns have been posted in the "Communication" feedback section under general comments. Please see there for a longer review. I encourage the authors to continue this work and polish the current draft.

---

Distill employs a reviewer worksheet as a help for reviewers.

The first three parts of this worksheet ask reviewers to rate a submission along certain dimensions on a scale from 1 to 5. While the scale meaning is consistently "higher is better", please read the explanations for our expectations for each score—we do not expect even exceptionally good papers to receive a perfect score in every category, and expect most papers to be around a 3 in most categories.

Any concerns or conflicts of interest that you are aware of?: n/a
What type of contributions does this article make?: Novel results

Advancing the Dialogue	Score
How significant are these contributions?	3/5

Outstanding Communication	Score
Article Structure	3/5
Writing Style	1/5
Diagram & Interface Style	3/5
Impact of diagrams / interfaces / tools for thought?	4/5
Readability	2/5

Remarks on communication

Below I first list some pros of the communication of this submission, followed by a list of concerns and a general review.

Some pros

• The first interactive graphic and text in the Background is well motivated and interesting to read
• The description and motivation of the article using the Data-visual correspondence is a great framing of the visualization techniques used here, especially when applied to machine learning
• After the Grand Tour technique is introduced and explored, the adversarial example exploration was compelling to see other uses of the developed visualization design

Major: Submission readability

While the submission is readable and the main ideas are accurately presented, it suffers from pervasive grammatical errors and uncommon word choices that negatively impact the communication of the research. This is my primary concern with the submission. The research here is interesting and I am excited to see it eventually published, but as is, it needs multiple rounds of prose iteration and improvement to meet academic publishing standards (Distill, or any other computer science conference).

Major: Grand Tour technique introduction

Given the emphasis on the Grand Tour technique, which is an interesting perspective on visualizing neural network activations, there is likely a better way to introduce and frame the technique with respect to modern techniques. Specifically, the phrase “somewhat-forgotten” feels self-defeating and could be improved to excite the reader about this technique. Furthermore, for readers that may not know of it, was it somewhat-forgotten for good reason? Or did it simply fall out of research popularity? Some discussion around this may be helpful to properly situate the technique in history.

Major: Better presentation of technical details

The technical details section would be better placed in the acknowledgements of the article. However, since it is quite long (somewhere between 1/4 and 1/3 of the article), another idea is that it could be highlighted with a banner description and hidden by default with an indicator to reveal upon a reader interaction (e.g., show/hide button).

Minor: Hero / banner interactive graphic

While the top-level graphic is eye-catching and hints at the techniques used in the paper, perhaps a caption and light annotation could help improve its message to readers. This would help push it towards substantial preview of the article, rather than only a nice looking animation.

Minor: Broken graphic

“The Grand Tour of the softmax layer” figure was broken: the image/points drop down menu did not update the visualization correctly, nor did the data instance positions did not update.

Minor: Miscellaneous

• Capitalize headings?
• “The math behind the two is simple.” Be careful with phrases such as these, since readers will come from many diverse backgrounds with different levels of mathematical fluency.
• Adding a zoom slider to all Grand Tour visualizations (not just the later ones) could be useful
• “vis” instead of “visualization,” although explicitly indicated in the submission, reads a bit informal

Scientific Correctness & Integrity	Score
Are claims in the article well supported?	3/5
Does the article critically evaluate its limitations? How easily would a lay person understand them?	3/5
How easy would it be to replicate (or falsify) the results?	4/5
Does the article cite relevant work?	3/5
Does the article exhibit strong intellectual honesty and scientific hygiene?	3/5

Comments on Scientific Correctness and Integrity

While there could be other citations to include to support the first paragraph of the introduction (e.g., work related to visualization and visual analytics in deep learning, neural network interpretability), the current text passes.

In the Discussion, I expected to see citations or relevant links/materials to corroborate “The trade-offs between small multiples and animations is an ongoing discussion in the vis community.”