added discussion

karulf-thesis/thesis-conclusion.tex

@@ -23,5 +23,21 @@ \section{Future User Studies}
 
 As mentioned above, while we can correlate successful task completion with improved usability, we would like to utilize a more direct metric. Researchers at the United States National Aeronautics and Space Administration (NASA) developed a metric for describing a human's workload while accomplishing a task. The metric, known as the ``Task Load Index,'' is computed from a user's perceived workload on six sub-scales: mental demand, physical demand, temporal demand, performance, effort, and frustration \cite{NASA_TLX}. Our revised user study would ask subjects to rate their workload on each of the Task Load Index's six sub-scales after each experiment. At the conclusion of the user study, we would ask users to answer the fifteen questions to determine the weighting of the sub-scales when computing the task load index \cite{NASA_TLX20}.
 
-Given the context of testing the control modes using the NASA Task Load Index, we decided to modify the subjects' tasks to more accurately represent our use cases. These new - more challenging - tasks would require a larger simulated world, more robots, and a different scenario. A weakness in the preliminary user study's scenario, is that the robots were able to complete their task immediately. This model does not match a use case in the real world, where most actions will require a non-zero amount of time to complete. Such a study, we believe, will show the benefits of mixed-mode interfaces across the whole population when even experienced users cannot realistically monitor all robots at once.
+Given the context of testing the control modes using the NASA Task Load Index, we decided to modify the subjects' tasks to more accurately represent our use cases. These new - more challenging - tasks would require a larger simulated world, more robots, and a different scenario. A weakness in the preliminary user study's scenario, is that the robots were able to complete their task immediately. This model does not match a use case in the real world, where most actions will require a non-zero amount of time to complete. Such a study, we believe, will show the benefits of mixed-mode interfaces, across the whole population, when even experienced users cannot realistically monitor all robots at once.
 
+\section{Discussion} % (fold)
+\label{sec:discussion}
+
+The results presented in the previous section show, unsurprisingly, that subjects who play video games or have prior experience controlling robots are better at performing the search task (ie. have lower completion, total neglect, and total idle times) with the RIDE interface than those who do not regularly play video games. While experience with RTS games seems to have no significant effect on the how well subjects perform the task, experience with first-person games clearly demonstrates an effect, decreasing all three times. First-person game experience caused subjects to use the first-person interface less than subjects who did not play first-person games, suggesting experienced first-person gamers recognize limitations of the interface.
+
+All subjects, regardless of experience, showed a preference towards the supervisory mode interface spending over 69\% of their time, on average, in this mode. The first-person mode was only used by a single subject during non-training runs. This leads us to believe limitations in our current implementation of the first-person are preventing adoption. Currently, the map is rendered as a texture on the ground plane in all modes, making it hard to see and correctly interpret at acute viewing angles. We hypothesize extruding the map into 3D space, effectively creating ``walls'' coming out of the floor, would provide a first-person user a much better sense of positional awareness with respect to the map. The idea of extruding the map to create 3D walls is similar to an interface by Bruemmer et al \cite{Bruemmer05turnoff}.
+
+The percentage of time spent in supervisory mode dramatically affects the times, with higher percentages correlating with much faster completion, and much reduced neglect and idle times. This suggests that an RTS interface is more appropriate than first- or third-person interfaces for this type of search task.
+
+Notifications have a similarly beneficial effect, making the task subjectively easier to perform, and causing the subjects to ask for help less frequently. Surprisingly, across our whole population, notifications did not have significant effect on completion time, total neglect time, or total idle time.
+
+However, the use of notifications did have a significant effect on subjects with no prior robot control experience, lowering completion time, neglect time, and idle time dramatically. We attribute this difference in effect to a flaw in our experimental design. We believe that the two-robot search task was not taxing enough for those who had controlled robots before. We hypothesize that these advanced subjects were able to manually monitor both robots at the same time, actively anticipating failures, making the notification system redundant. More inexperienced users, on the other hand, might not notice a robot in need of help when attending to the other robot, leading to increased total neglect and idle times. In this situation, notifications serve a critical function.
+
+In conclusion, we believe that mixed-mode video-game-based interfaces, such as RIDE, offer an effective way to control large numbers of mostly-autonomous mobile robots. Our initial studies suggest that an active notification system allows inexperienced users to more efficiently control teams of robots, reducing neglect and idle times. We believe that this result will also apply to more experienced users in situations where they are sufficiently overloaded, or where unexpected events happen frequently.
+
+% section discussion (end)