Review of "Self-driving Lego Mindstorms Robot"

[moorepants]

Independent Review Report

.. note:: Please be aware that all reviews are made public including
the reviewer's name.

Reviewer: Jason K. Moore

Department/Center/Division: Human Motion and Control Lab, Mechanical Engineering Department

Institution/University/Company: Cleveland State University

Field of interest / expertise: multibody dynamics, biomechanics, control
systems, system identification

Country: USA

Article reviewed: Self-driving Lego Mindstorms Robot

INSTRUCTIONS

Please read the submitted article and fully complete this form. Since we don't
have a copy editor, we also request that you annotate the PDF [1]_ to highlight
typos, formatting issues, and grammatical mistakes.

The goal of the review process is two-fold. First, it guides authors in
improving their papers and, secondly, ensures that published works are of a
professional academic standard.

Research in science and engineering increasingly relies on software for
data processing and management as well as theoretical exploration. However,
the effort necessary to develop this software is rarely recognized as having
the same academic worth as other aspects of the research. These proceedings
are, at least in part, intended to address this shortcoming.

An article focused on software development necessarily differs from the
standard scientific article with respect to format. For instance, it is
unlikely to have the same sections (i.e., introduction, methods, results,
conclusion). You may therefore have to rely on other factors to decide whether
the paper sets a high enough standards as an academic publication.

Please note that, while reviewers' recommendations regarding a paper's
suitability for publication are seriously considered, the final decision rests
with the proceeding editors.

.. [1] We recommend the free version of PDF XChange Viewer <http://www.tracker-software.com/product/pdf-xchange-viewer>__ for
Linux (Wine) and Windows. Under OSX, annotation is provided by Preview
as well as Skim <http://skim-app.sourceforge.net/>__.

GENERAL EVALUATION

Please rate the paper using the following criteria (please use the abbreviation
to the right of the description)::

below doesn't meet standards for academic publication
meets meets or exceeds the standards for academic publication
n/a not applicable

• Quality of the approach: below
• Quality of the writing: below
• Quality of the figures/tables: below

SPECIFIC EVALUATION

For the following questions, please respond with 'yes' or 'no'. If you
answer 'no', please provide a brief, one- to two-sentence explanation.

• Is the code made publicly available and does the article sufficiently
  describe how to access it? We aim not to publish papers that essentially
  advertise proprietary software. Therefore, if the code is not publicly
  available, please provide a one- to two- sentence response to each of the
  following questions:
  • Does the article focus on a topic other than the features
    of the software itself?
  • Can the majority of statements made be externally validated
    (i.e., without the use of the software)?
  • Is the information presented of interest to readers other than
    those at whom the software is aimed?
  • Is there any other aspect of the article that would
    justify including it despite the fact that the code
    isn't available?
  • Does the article discuss the reasons the software is closed?

Yes, the code is publicly available and relies on a variety of open source
packages. The software can be downloaded from the github link provided in the
paper. The Android image capture library may not be open source.

• Does the article present the problem in an appropriate context?

Yes.

Specifically, does it:

• explain why the problem is important,

Yes, he gives background on self-driving car research to show the need.

• describe in which situations it arises,
• outline relevant previous work,

Maybe, only a couple of papers are cited about self-driving vehicles and
techniques. Much of the literature is not present and no detailed commentary on
how the method presented in the paper compares to other approaches.

• provide background information for non-experts

Yes, the paper seems to be written with a non-expert audience in mind.

• Is the content of the paper accessible to a computational scientist
  with no specific knowledge in the given field?

Yes.

• Does the paper describe a well-formulated scientific or technical
  achievement?

No, the paper doesn't propose a hypothesis or follow the scientific method. It
is more like a report on how to use several software libraries to accomplish a
task rather than a well-formulated scientific achievement. I guess it can be
called a "technical achievement" because the author achieved his goal of
reproducing another's work with different software and hardware.

• Are the technical and scientific decisions well-motivated and
  clearly explained?

No, the motivation only seems to be to replicate previous work. The reasons for
choosing the hardware, software, and parameters for both are not explained at
all. It seems as if the author just used informed guesses at values for the
neural network, for example.

• Are the code examples (if any) sound, clear, and well-written?

Yes, but it'd be nice if they followed PEP8 standards for readability reasons.

• Is the paper factual correct?

Yes, the method and results seem to be factually correct.

• Is the language and grammar of sufficient quality?

Yes.

• Are the conclusions justified?

No, because there are no conclusions. The paper simply describes how something
is done. Most scientific works explain what the conclusions are and make some
reasoning on why those conclusions are true. The conclusion here seems to be
simply that someone else's work can be replicated with different hardware and
software while using the same methods.

• Is prior work properly and fully cited?

No. The blog post that described the work that is being replicated is cited
with a URL and a few academic papers are cited, but the wide berth of work on
self-driving vehicles and neural networks has been ignored.

• Should any part of the article be shortened or expanded? Please explain.

Yes. The prior literature on the subject should be expanded and comparisons in
this method and others is needed. Furthermore, there should be some scientific
discourse on the details of this method along with quantitative measures
describing the performance of this technique so that comparisons can be made to
other software, hardware, and methods.

• In your view, is the paper fit for publication in the conference proceedings?
  Please suggest specific improvements and indicate whether you think the
  article needs a significant rewrite (rather than a minor revision).

This paper presents the replication of a "self-driving" robot vehicle
implementation that utilizes a trained neural network to follow a specific
course using visual inputs to control the vehicle's driving motors. From what
is written, it seems that using a Python based software stack and the Lego
robot kit, that prior work can be replicated. But the article does not exhibit
the depth that other quality scientific articles on this subject offer. The
reader is simply instructed in the how, i.e. the method, of implementing this
system using a very specific selection of software packages. Little to no
information is provided that gives the reader technical information on the
capabilities of this method, particularly not for comparison purposes to other
methods. No hypothesis or claim is made nor any proof to back it up the missing
claim. The article seems more akin to a undergraduate lab tutorial that simply
shows the student how to do something, but misses the "why" portion that
generally makes a contribution interesting and publishable for the scientific
community. I think this article could be transformed into a valuable scientific
contribution if these things were changed/added:

1. A proper literature review on other methods. At the minimum, this could
   detail other software libraries with these capabilities and at the maximum
   this could include comparison to other methods of autonomously controlling a
   vehicle.
2. A statement, hypothesis, or claim about what makes this method
   special/difference and the proof to back it up. If this is simply a
   replication study of previous work using different methods, then the claims
   from the previous study should also be proved by the method presented in
   this paper along with detailed quantitative comparisons of how well the
   other method was replicated.
3. More technical detail on the method. If comparing software, we need to know
   things like how easy it is to use, how fast it is, how robust it is, what
   are the limitations, etc all with respect to other available software. The
   technical details of the hardware are also important so that we no its
   advantages and limitations with respect to other methods. If comparing
   algorithms (neural nets, etc) then we need to know more detail about the
   methods and why the parameters you chose are good and what they mean.
   Explanation of the neural net framework you chose and why would be helpful.
4. Give accurate and precise results. Simply saying that your vehicle completes
   the course "about 2/3rds" of the time is not science. You also need
   dimensional descriptions of the course, the vehicle, and metrics on how bad
   or good it actually performs. No one can compare their vehicle and
   implementation to yours if this isn't provided. We also have no idea if you
   actually replicated the prior work because there are no quantitative
   measures.
5. The tone and grammar of the paper resembles a blog post as opposed to a
   scientific article. I'm not opposed to having more personal writing, but it
   needs to be justified and it needs to contribute to the understanding of the
   article. As it stands, this would be a fine blog post but it is quite far
   from an average scientific article.