Usability Tests

Table of Contents

• Usability Testing Plan
• Usability Testing Results - Round 1
• Usability Testing Results - Round 2
• Usability Testing Results - Round 3

---

Usability Testing Plan

The test plan addresses the how, when, where, who, why, and what of our usability testing. We've documented our approach to set a consistent framework that will serve us not only in this initial round of testing but in all subsequent rounds. Using the same test plan structure across multiple rounds allows us to:

• Maintain methodological consistency, making results comparable over time
• Onboard new team members or stakeholders more efficiently
• Track how user needs and pain points evolve as the product develops
• Build institutional knowledge about what works (and what doesn't) in our testing approach

The following sections detail the full usability testing program for VeloSim. On this Wiki page you can also find results from past rounds of testing.

Current goals and objectives

• Evaluate whether the current features meet the needs of all users
  • particularly BIXI dispatchers (the domain experts) as the primary target group
• Identify gaps between what end users need and what is currently provided
• Discover usability issues and pain points in existing features
• Prioritize future requirements based on user feedback and observed behaviour

Current research questions

• Current System Perception:
  • What are users' overall attitudes/satisfaction with the current system?
• Feature Assessment:
  • What tasks are users trying to accomplish that the current software doesn't support?
  • What features do users expect to find but couldn't?
  • Which existing features are intuitive vs. confusing or difficult to use?
• Workflow & Efficiency:
  • Where do users get stuck or frustrated in their workflows?
  • What would make users more efficient or productive in their work?
  • What workarounds, if any, are users creating to compensate for missing functionality?
  • How well does the software align with users' mental models of the problem domain?
• Unexpected behaviour in a production system:
  • Did end users encounter any issues, bugs, or unexpected behaviour?

Participant characteristics

These personas trace back to the types of users that we've documented on the Personas Wiki page.

Participant/Persona Type	Experience Level	Notes
Dispatcher	Domain experts and friends/family	Domain experts when available; friends/family debriefed with a script
Scenario Administrator	Simulated by friends and family	Debriefed with a script

Test method and setup

Test Design:

• Type: Moderated usability testing (formative evaluation)
• Estimated duration per session: 30 minutes (actual duration as of round 1 usability testing: 20 - 35 minutes)
• Structure: Think-aloud protocol with task completion

Environment:

• Platform: Production website (https://velosim.app)
• Devices: Desktop and laptop computers
  • While VeloSim's interface is responsive and supports mobile and tablet devices, the primary user base accesses the product on desktop and laptop computers. Accordingly, usability testing will focus on these platforms.

Data Collection:

• Screen capture and audio: OBS Studio
• Additional feedback: Post-test form via Google Forms

Moderator Protocol (click here to reveal dropdown summary)

1. Open the testing script in a browser window
2. Open VeloSim production environment (https://velosim.app)
3. Create a new non-admin user account for the participant (record credentials)
4. For Dispatcher personas: Download the test scenario file
5. Open the feedback form
6. Begin screen recording with OBS Studio
7. Hand computer to participant and instruct them to read the introduction and work through tasks
8. Observe and note any criticisms or issues during the session
9. Ensure participants complete checkpoint questions in the feedback form
10. After completion, stop recording, name video file with participant username, and upload to designated Google Drive folder

Participant task lists

Dispatcher Persona Tasks (click here to reveal dropdown summary)

Context: Participants role-play as BIXI task dispatchers who must manage battery swap operations. They assign tasks to resources, which are drivers that have a vehicle assigned to them. Drivers without a vehicle wait at the BIXI headquarters (HQ) until one becomes available. You'll be working on a simplified version of the BIXI network, with only two resources and two stations to manage.

Tasks:

1. Log into VeloSim using provided credentials
2. Import a pre-configured simulation scenario
3. Start the simulation
4. Zoom in to view station icons
5. Speed up the simulation
6. Wait for tasks to appear at stations
7. Pause the simulation
8. Click on stations to check existing tasks
9. Assign a task to a driver
10. Resume the simulation
11. Assign remaining tasks to the other driver
12. Reorder the task queue for a driver with multiple tasks
13. Explore additional features: unassigning tasks, dragging tasks to drivers

Checkpoints:

• After Tasks 1-2: Rate ease of login and scenario loading
• After Task 12: Rate ease of time navigation, task assignment, and task reordering
• Final: Complete general feedback section

Scenario Administrator Persona Tasks (click here to reveal dropdown summary)

Context: Participants create simulation scenarios by defining stations, drivers, vehicles, and scheduled tasks using the Scenario Editor.

Tasks:

1. Log into VeloSim using provided credentials
2. Navigate to Scenario Editor and create a new scenario
3. Draft a scenario following a detailed specification including:
   • Three stations with specific locations, initial tasks, and scheduled tasks
   • Two drivers with shift times and lunch breaks
   • Two vehicles with starting locations and battery inventory
4. Explore additional features: saving scenarios, creating duplicates, exporting to local computer, editing scenario titles, creating new versions vs. overriding existing scenarios

Checkpoints:

• After Task 3: Rate ease and effectiveness of Scenario Editor, suggest improvements, evaluate error message clarity
• Final: Complete general feedback section

Data to be collected and evaluation measures

• Quantitative data
  • Likert-scale questions about ease of use for feature set
  • Task-specific ease ratings (login, scenario loading, time navigation, task assignment, task re-ordering, scenario editor usage, error message clarity)
• Qualitative data
  • Observed pain points, confusions, and delights
  • Feature requests and suggestions
  • Positive feedback and what's working well
  • Open-ended feedback on overall system experience and suggested improvements

All quantitative and qualitative feedback will be collected through a standardized post-test Google form.

Report contents and presentation

• GitHub wiki page containing all our Usability Testing information
  • Summarizes the background of the study
    • including the goals, methodology, and participant characteristics
  • Present findings for original questions to investigate
  • Give user feedback results and discuss specifics (appropriate to data)
  • Suggest follow-up research
• SOEN 490 Release Presentation videos
  • Summarize usability testing plan and our results
  • Discuss how feedback was incorporated into design and implementation

Usability Testing Results - Round 1

We have maintained an ongoing collaborative feedback loop with our stakeholder throughout development, incorporating their input at multiple touchpoints including design mock-ups, live production testing, and backlog planning. Reaching our second release milestone provided an opportunity to conduct our very first round of structured usability testing. From January 13 to January 17, we engaged friends and family as test users, guiding them through a standardized bilingual script that evaluated core workflows of real BIXI dispatchers. We also had the opportunity to get feedback from 1 domain expert at BIXI (part of the core user base following the end of Release 3).

We collected both quantitative usability metrics and qualitative feedback to meet our initial objectives and answer the research questions described below.

Summary

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level	Notes
Dispatcher	12 (minimum)	22	Domain experts and friends/family	1 domain expert, 21 friends/family debriefed with a script
Scenario Administrator	3 (minimum)	9	Simulated by friends and family	Debriefed with a script

Note: Only 1 domain expert was available to participate during the Release 2 testing window.

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

Dispatcher Task Completion Time Results

Task	Description	Min (s)	Max (s)	Average Time to Complete (s)
T1	Login	2	15	6.050
T2	Import	2	40	7.895
T3	Start Sim	1	5	2.105
T4	Zoom to See Station Icons	2	8	4.316
T5	Speed up Sim	2	23	4.737
T7	Pause the Sim	1	3	1.263
T8	Click on stations to see tasks	1	4	1.473
T9	Assign a task to a driver	2	28	8.158
T10	Resume the sim	1	2	1.158
T11	Assign all remaining station tasks	5	70	23.526
T12	Reorder driver's task queue	2	29	5.474

Note: Task 6 involved waiting for tasks for appear at stations, where the user simply remained idle for a few seconds as the simulation sped up to their preference. For this reason, it was not considered in the time results.

Scenario Admin Survey Results

Likert Scale Questions (usability of scenario editor, helpfulness of error messages):

Scenario Admin Task Completion Time Results

Metric	Task 1: Log in	Task 2: Create from Template	Task 3: Edit Scenario
Mean	5.11s	5.13s	784.22s
Sample Size	n=9	n=8	n=9

Note: 1 screen recording was not available, and 1 participant failed to properly carried out task 2. This is the reason why the sample sizes differ for these results.

General Survey Results

General User Feedback Compilation

Below, feedback is split by type / source / feature group. Feedback from the domain expert is placed in its own category, since this feedback takes higher priority than others'.

Feedback that came from a domain expert (click here to reveal dropdown summary)

• The task assignment confirmation is important to have, but becomes annoying if you can't mass-assign tasks
• Multiple tasks should be selectable via both drag-to-select and control+click to select
• The vehicle icons should have driver names to distinguish them
  • In the real BIXI network, there are multiple task types - so we would also want the vehicle name displayed for each resource since different vehicles perform different tasks
  • Vehicle names would be displayed over the vehicle icon and in the resource bar
• The vehicle icon should be dynamic to display information related to resource state and direction of travel
• The selected item state should stay on station after a task assignment
  • This is mostly because, in the real BIXI network, there are multiple task types
  • In the context of VeloSim, if mass-assignment is implemented, there is less concern about switching state on assignment
• Hard to see what stations are his priority, believes clusters would lead to a clean map but wouldn't help him find his priorities
• There should be the option to drag scenario files into the scenario editor to import them
• User isn't confident about being able to locate task priorities through the map alone
  • Hard to find stations with many tasks when looking for larger numbers in a sea of numbers
  • Believes a universal tasks bar would help in this regard
  • In the context of VeloSim (only battery-swap tasks), user doesn't care about stations with 0 tasks
  • In the real life BIXI network, where bike allocation tasks exist, user would care about all stations
• User believes it would be nice to be able to assign without dispatching (out of scope for our project as decided by stakeholder)

Feedback on the simulation page (click here to reveal dropdown summary)

• The icons, especially that of HQ, are confusing (8)
  • One proposed an icon legend
  • One proposed the ability to click HQ
  • This has been identified to be an issue of the usability test pre-amble not introducing the notion of HQ
• The process of assigning a task is difficult to learn (6)
  • One proposed putting a note in the driver's selected item bar
  • One proposed including a prompt to explain the task-assignment feature
  • One proposed highlighting the cars on the map and driver names in resource bar when a task enters drag mode
  • There is no affordance for dragging tasks, and no indication that you can drag them to the resource bar
• The automatic switch from station to vehicle as the selected item on task assignment is annoying (5)
• There should be a way to select and mass-assign tasks (4)
• The vehicle icons on the map should display the driver names or vehicle id above them (3)
• The task re-ordering UI should have friendlier bounds (3)
• There is nearly no visible distinction between 2 drivers in the same location (same station) due to overlap (3)
  • It is hard to select assign tasks to overlapping vehicles
  • One proposition is to enlarge the car icon being hovered over
• There should be a way to select tasks in the same way as stations and vehicles (2)
  • There is no way to track that a task is assigned to, from the selected stations bar
• It is difficult to select a station when a driver is at that station due to icon overlap (2)
• There shouldn't be a confirmation dialog for task assignments (2)
• The meaning of the numbers on the HQ widget isn't clear (2)
• Confusion related to not being able to re-assign / re-order in-progress tasks (2)
• The selected item bar can hide too much of the map (2)
  • The task box is unnecessarily long
  • One proposition is to make selected item bar translucent / invisible on task drag state
• There should be a built-in tutorial or documentation to reference on the site (2)
• The dots representing stations when zoomed out should be bigger (2)
• There should be on-icon-hover-enabled tooltips for quick station / driver info (2)
• The sidebar on the simulation page should be removed or automatically minimized due to poor use of space (2)
• The large vehicle icon overlaps station dots when zoomed out far enough
• The simulation should be more realistic (traffic, weather events, construction, accidents)
• The car icon should reflect a bird's eye view of a vehicle
• The status of drivers should be included in the resource bar
• The selection of an entity should center the map on that entity
• The in-progress state should be split into two distinct states (on route and servicing)
• The process of re-ordering tasks is difficult to learn
• There is no easy way to tell what driver a task is assigned to
  • Could put a badge / label on the task item in the station's selected item card
  • Universal task sidebar that could include details about which driver has been assigned that task
• Desire to easily see how many tasks require assignment at each station, rather than (only) how many tasks total
• Desire to filter resource bar by driver name instead of just vehicle id
• Unsure what the number above the station represented
• Desire for universal tasks bar
• Want some way to know a driver is taking their break
• Would like marker clustering
  • Each cluster displays a number indicating how many stations it contains
• Would like sim-resumability
• Thought the arrow keys would speed up the sim
• Would like to know what the route would be before task assignment
• Needs to have more visual queues to understand what the simulation is doing

Unless stated otherwise, every point above was only mentioned once.

Feedback on the scenario editor page (click here to reveal dropdown summary)

- The use of the JSON format for defining scenarios is problematic (10) - Poor clarity, poor readability, difficulty troubleshooting validation issues, too text-heavy - Proposed alternatives include "something more visual" (map-based), or something form-based - One alternative is a helper modal to interact with the JSON (button to add driver, vehicle, station) - One suggestion was to offer support for a different format than JSON - The scenario editor lacks functionality expected in modern text editors (2) - Examples include tab-friendliness, line numbers, syntax coloring, and collapsable fields - The scenario editor should allow imports by dragging files in (2) - The scenario should save automatically when pressing *Start simulation* - The fonts used for field inputs differ from the fonts used for field labels - The font for the description field input shouldn't look like code (mono) - The JSON scenario fields should be collapsible - The import and export buttons should be nearer to one another - The save and export buttons should be the same colour - the import and new buttons should be the same colour - The scenario should be immediately editable upon selection - The scenario validation error messages are not helpful enough - There should be a way to make the scenario description field textarea bigger - There should be a dedicated *Validate scenario* button - The use of red for the selected scenario is confusing since red usually indicates something bad - The placement of the scenario description field before the scenario content field is unnatural

Unless stated otherwise, every point above was only mentioned once.

Feedback that was positive (click here to reveal dropdown summary)

Most feedback was critical of the application, but a few users highlighted things they liked about VeloSim:

• The flexibility in how you can assign tasks (drag to icon or resource bar row) accommodates different needs (2)
• The ability to click on stations / vehicles to see more information about them is helpful
• The visualization of a driver's route is helpful
• The confirmation dialog that appears on attempted task assignment prevents mistakes when dragging over vehicles that are near each other / overlapping
• The New button template in the scenario editor serves as a helpful guide

Unless stated otherwise, every point above was only mentioned once.

Feedback on the login page (click here to reveal dropdown summary)

• The login page should allow and ignore white-space around the username

Every point above was only mentioned once.

Feedback on admin-only users page (click here to reveal dropdown summary)

Some users were accidentally given an admin account to perform usability testing. The following is feedback they shared on the users dashboard.

• The Previous and Next buttons should remain at a fixed position when paginating
• The users table should be sortable by specified column
• The Id header of the users table should be labelled ID
• The users table should be filterable
• Admins should be able to see all currently running simulations, and be able to stop/pause them
• The ability to reset your own password from the sidebar should be more obvious
• The reset-password dialog should omit the obvious indication to "Click update when you're done"
• The create-use card should disappear after successful user creation

Every point above was only mentioned once.

Feedback that conflicts with project requirements (click here to reveal dropdown summary)

Some feedback either contradicts how the system is intended to work and can't be addressed without conflicting with project requirements.

• The numbering of tasks (their id generation) should be on a per-station level, not global
• There should be a built-in algorithm to automatically assign tasks to drivers already at a station
• The task lists belonging to stations should be re-orderable
• The tasks at a station should have a service priority

Every point above was only mentioned once.

Feedback on the usability test itself (click here to reveal dropdown summary)

Some feedback pertained to how the usability test was administered:

• Some didn't know what a "resources bar" was supposed to be (2)
• The test instructions should be clearer / simpler (2)
• The survey should not have a second open-ended question for scenario admins
• The usability test should introduce the notion of HQ in the beginning
• The stated user goal of ensuring efficient task assingments that save time and fuel is not satisfiable given the simplicity of the simulation used
• The dispatcher test should specify that the simulation is expected to have two stations
• The test instructions should not be in a separate tab
• The test should communicate the fact that you must zoom in to see the full station icons
• The tables in the scenario admin test should have titles that match the scenario attribute field names
• The system should be demoed to the user prior to attempting the usability test

Unless stated otherwise, every point above was only mentioned once.

Bugs discovered during usability testing (click here to reveal dropdown summary)

• Task assignment via dragging to the resources bar would highlight multiple driver rows instead of just one
• A one-time 502 error was encountered in Firefox
• Clicking on the current scenario after it has been edited shows a "Unsaved Changes" card
• User search bar in the users dashboard does not work when looking at a non-first page

Unless stated otherwise, every point above was only mentioned once.

Key Takeaways

• The simulation page needs its own mechanisms to explain its various parts to users (e.g. a legend for icons, task assignment mechanics, etc.)
• The simulation page needs more affordances for possible actions and clearer feedback about task-driver relationships
• The simulation page needs to provide more options for flexible and efficient task assignment
• Vehicle icons on the map need more identifying characteristics (i.e. driver name, vehicle ID, direction of route)
• Entity icons on the map generally suffer from overlap-induced visibility issues
• The scenario editor page needs to have more options for non-technical users
• The scenario content text area needs to be more feature rich due to the size of content being worked with there
• The scenario editor page should allow imports via dragging since this design pattern is commonplace

Action Items & Issues Created

Simulation Page Improvements

• Enable mass-assignment of tasks #586
• Add driver names above vehicle icons #587
• Add resource state information and direction of travel to vehicle icons #588
• Add resource state information to resource bar #589
• Add resource state information to selected item bar #547

• Maintain selected item state on task assignment #590

• Reduce visibility of 0-task stations #591

Scenario Editor Page Improvements

• Offer per-entity forms for scenario manipulation #592
• Add line numbers to the scenario content textarea #593
• Add drag and drop support for scenario importing #456

Bug Fixes

• Resource bar task-assignment hover persists #583
• Unsaved changes card appears when attempting navigating to current scenario #584
• Users search bar breaks past first page of users #585

Usability Test Improvements

• Introduce the notion of resources as driver-vehicle pairs in the dispatcher usability test script
• Introduce the notion of HQ in the introduction portion of each usability test script
• Remove open-ended question from scenario admin section of survey to prevent response duplication
• Remove the stated user goal of ensuring efficient task assignments that "save time and fuel" (not really applicable to the scenario used for usability testing)
• Add information pertaining to the total number of stations in the dispatcher usabilty test script
• Update the table names in the scenario admin test to match the scenario attribute field names

Usability Testing Results - Round 2

Summary

Our second round of usability testing took place on February 20 with a new domain expert. No other participants were available for Round 2.

• Duration: 30 minutes
• Test language: French usability test script was used, which surfaced localization gaps.

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level
Dispatcher	1	1	Domain expert

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

General Survey Results

Domain Expert Feedback (Round 2)

• French localization: the test revealed there are currently no French control options in the website.
• Multi-select stations: participant requested the ability to multi-select one or more stations (multi-select station story) to assign or inspect many stations at once.
• Route planning for off-shift drivers: participant noted you cannot plan routes for drivers who haven't started their shift, but was unsure whether other domain experts would want this functionality.
• Clustering: participant likes the idea of clustering to reduce "missing" tasks (stations that should be serviced but are overlooked).
  • Edge cases: clustering needs a notion of cluster "frontiers" or a buffer distance to avoid grouping stations across natural boundaries. To address this, a suggestion was to add a distance threshold/buffer by geographical areas to prevent inappropriate clustering.
• Participant mentioned that the UI should switch focus to the driver for tasks assigned to them.
• Exact stop order: participant wants to see the exact order of stops when viewing a route; to address this, one idea is to display stations on hover for tasks in the task list and add a clear stop-order indicator in the route view.
• Route display toggle: our offering to toggle between showing a single route vs all routes.
  • A complementary idea is (when in all-routes-showing state) to consider auto-hiding all other routes when hovering a vehicle.
• Asked about automatic dispatching (as an alternative to running the simulation manually and interacting with the UI). The team explained that scripts can be used to interact with the VeloSim API in order to carry out automatic dispatching.
• Asked about average time to carry out a task: the team explained the current simulation reporting feature that's being implemented along with its specifics.

Key Takeaways

• Localization: lack of French UI controls affected the test and should be addressed for francophone users.
• Multi-station operations: following its addition after the first round of usability testing, multi-select support is a high-value feature for domain experts managing many stations.
• As mentioned above, the participant mentioned that the UI should switch focus to the driver for tasks assigned to them. During Round 1 usability testing, mass task assignment was not yet available: tasks could only be assigned one at a time through the manual simulation UI. So, participants preferred maintaining their current context rather than having the interface automatically shift focus. Now that mass task assignment has been introduced, the workflow has changed: assigning multiple tasks at once naturally shifts the user’s focus to monitoring the driver’s workload. In this updated context, automatically switching focus to the driver better supports the user’s mental model and is more efficient than the behaviour implemented after Round 1.
• Clustering is a good idea but must respect geographic boundaries; naive clustering can create problematic assignments in edge cases.
• Route visibility: users need clearer, per-task and per-route visual cues (stop order and single-route focus on hover).
• On a 5-point Likert scale, the ability to re-order tasks received a 3, indicating moderate satisfaction and clear room for improvement. We acknowledge this feedback and will further investigate the underlying usability challenges to identify opportunities to improve clarity, efficiency, and the desired UX in task reordering.

Action Items & Issues Created

• Introduce localization with French UI controls (labels, buttons, help text).
• Implement multi-select stations (ideas: selection via ctrl+click, shift+click, and drag-to-select) — create a "multi-select station" story.
• Investigate support for planning routes for drivers not yet on shift; consult stakeholder and other domain experts for desirability.
  • After discussion with stakeholder, this should not be supported.
• Investigate ways to enhance the process of re-ordering tasks
• Implement clustering with a configurable buffer/distance threshold to avoid crossing natural boundaries.
• Allow UI switches to the assigned person/resource when tasks are assigned in bulk.
• Show order of stations that a driver is on route to service

Usability Testing Results - Round 3

Summary

This time around, we decided to alter the format of the usability tests. We found in previous rounds that having the user switch between the VeloSim application and a survey form isn't ideal for preserving their train of thought / general focus. Instead, the moderator would fill the form as they orally gave users tasks to accomplish. Tasks are purposefully open-ended to observe what approach users take to accomplish them (and with what amount of difficulty). This allows us to evaluate the overall discoverability of specific features. Users are asked to "think aloud" during the process.

Tasks given to participants in this usability testing round are:

• Single Task Assignment: Select a station with more than one task and assign a single task from that station to a resource.
• Multi-Task Assignment: Select another station with more than two tasks and simultaneously assign all its tasks except for one to a resource.
• Single Station Assignment: Simultaneously assign an entire station’s worth of tasks to a single resource.
• Multi-Station Assignment: Simultaneously assign multiple stations to a single resource.
• Cluster Assignment: Zoom out and assign a cluster to a resource.
• Task Re-Ordering: Re-order a resource's tasks.
• Task Unassignment: Un-assign one or more tasks from a resource.
• Task Reassignment: Assign tasks belonging to one resource, directly to another resource (without first un-assigning them).

Note that this round of usability testing entirely focuses on the dispatcher user persona. This is intentional, as the dispatcher experience is deemed of highest interest to the stakeholder (and is considered to have the most area for improvement at this time).

Task Approach Summary

Most tasks given to participants in this test have more than one way in which they can be accomplished. The approach taken by the participant has been recorded in every case, and the results are described in the pie-charts below. These results give an impression of a feature's discoverability and intuitiveness (relative to other features that enable the same functionality).

Multi-Task Assignment

Single Station Assignment

• Many participants did not know you could drag entire station.

Multi-Station Assignment

Task Difficulties

• Note that the low score for single task assignment has been observed to be due to an initial barrier in understanding that dragging is the primary mechanism by which all forms of task assignment occur. The following objectives were much easier once the dragging mechanism was learned.
• Note that the low score for the re-assignment task is due to a discovered bug (or simple lack of functionality) that prevents grouped tasks (the collapsed view of the selected driver bar) from being assigned.

User Feedback Compilation

Task Assignment Feedback

• Drag and drop not immediately intuitive (8)
  • The user had no indication or affordances.
  • Note it was really the initial understanding to drag and drop that was annoying
• Users preferred using a combination of available options for multi selecting tasks (5)
• Expected/would like a multi select checkbox style option for selecting tasks (2)
• modal for confirming assignment blended in and was not obvious (1)
• tried dragging tasks to the HQ widget instead of resource bar (1)
• Tried dragging tasks to station icon. (1)
• When starting the drag-to-select-indidivual-tasks, if its not started on the tasks, it doesn't select properly (1)
• Preferred using a combination of available options for multi selecting tasks (5)
• Expected/would like a multi select checkbox style option for selecting tasks (2)
• Would like a way to know visually from the map that all of a stations tasks had already been assigned. (1)
• thinks it would be good to have a button beside the tasks at a station to assign, and then be given a drop down of drivers to assign to (1)

Station Assignment Feedback

• Prefers dragging station to drivers over indivual task assignments (20)
• No affordance for being able to drag entire stations (13)
• Tried dragging driver to station instead of vice-versa (1)
• Interpets dragging a station to mean the station is being displaced, not assigned (1)
• Shift Click was expected to work for stations (3)
• Right click drag lacks precision for selecting stations (5)
  • Some noted that this was fine givne the ctrl click option and could be used in combination with right click drag

Cluster Assignment and related Feedback

• Expected the ability to right click drag for multi selecting clusters (6)
• Didn't expect clusters to break on click / thought they would be selectable (3)
• When hovering over a task, the cluster should highlight. (2)
• A user mentioned that they would prefer sector based clusters or specific "regions" on the map with task count information(1)

Task Updating Feedback (eg. reordering and/or unassigning)

• Need to show servicing on collapsed view & block dragging for that group (5)
• User tried re-ordering tasks at a station
• tried to re assign a task that was already being serviced. No warning appeared. (4)
• Assigned tasks could have more info such as the assigned driver as opposed to the simple 'x' (1)
• wants program to give advice when you're re-ordering, to prevent you from making inneficient decisions (1)
• expected to see both drivers side by side in the selected item bar on re-assignment (1)
• A user wasn’t sure originally that the 'x' represented unassign. The user tried clicking on the resource bar assuming more details would pop up about a task(1)
• User unintentionally unassigned multiple tasks at the same station when in the "grouped by station view", expecting to only unassign 1 task (1)
• User expected to be able to multi select (eg. ctrl click) multiple tasks at once to unassign them(1)

General UI feedback

This sections includes feedback related to the simulation UI and may not be directly related to the tasks performed during this usability test but are worth considering.

• The drivers starting at HQ is a bit tricky because I needed to take some time to notice their shape. (5)
  • Starting at HQ doesn't help
• Clicking on a task should move the map to center on that task's station (4)
• Users would like the task-hover-station icon to be bigger, more noticeable.(2)
  • This specifc ask is mentioned twice HOWEVER, many user's did NOT notice the hover effect until they were told about it. Should likely make this more noticable.
• The start simulation button should be disabled if no scenario is selected (1)
  • Technically, the scenario editor was out of scope for this usability test, but this issue was noticed regardless
• Complained about no legend of colors (1)
• Accidentally selecting text in components like selected item bar can be annoying (1)
• A driver's future route coloring (which have a lower opcaity) seem to confuse users, possibly since the opacity of the "future" routes are still too high and not distinguishable enough from the current route
• The ability to move the driver dialog box would be nice whe trying to view certain parts of the map that it may be blocking (1)
• Thinks double clicking driver in resource bar should center map on resource (1)
• Ideally, when selecting a vehicle or a task, the camera should lock onto what is selected(same as above?) (1)

General suggested feature/Improvements

This section includes features which users suggested which aren't directly related to the tasks performed in this usability test. They are however, worth considering for future development.

• When hovering over a task (or grouping) in a driver's task list, the cluster should highlight (2)
• User would prefer that a title like 'driver' would appear above the list of drivers. (1)
• would like interactive tutorial (1)
• Would like ETA for driver arrival times (1)
• User would prefer to use "Voice commands" to interact with the simulation rather than learning to use the UI. (1)
• User suggested that, when highlighting a task, bold the route to that task regardless of whether or not its the immediate next ask to be serviced (1).

General Positive feedback

• Likes drag and drop / found it intuitive (5)
• Resource bar useful when driver can't be found on map (1)

Collapsed vs Expanded Task Views

• 18 participants explicitely prefer the grouped (collapsed) task view, whereas 3 explicitely preffered the expanded task view
  • It is more clear where tasks are from in the collapsed view (2)
  • The meaning of the task ids in the expanded view isn't obious (2)
• Can't re-assign in expanded view (17)
  • 2 participants accomplised the Task Reassignment objective by dragging the station (or from the station's task list) to effectively re-assign (2)
• There is value in having access to both task views (8)
• Can't differentiate between collapsed / expanded task views (3)
  • Wishes the collapsed view used a Stations label instead of Tasks label above the list (1)
• Wishes to be able to see collapsed / expanded view at the same time (1)

Assigning via Map vs via Resource Bar

• 16 participants explicitly prefer assigning via the resource bar
  • Difficult to keep track of drivers spatially (10)
  • Easier distinguish between drivers, especially when they are near one another (or are ovelapping) on the map (e.g. at HQ) (3)
  • Easier to see what drivers are idle (3)
  • Easier to dish out tasks randomly (2)
  • Drivers move fast on high speed, making it difficult to assign to them (1)
• 8 participants explicitly prefer assigning via the map
  • Can't tell where drivers are / where they are going from the resources bar (8)
  • Generally don't have to drag as far, since the resource bar is on the far right (2)
• Didn't think you could drag to resources bar (3)
• Didn't think you could drag to resource on map (2)

Domain Expert Feedback (Pablo)

• Prefers the right-click+drag approach to select tasks
• Expects right-click+drag approach to work on the driver task list (however, this functionality is already supported)
• Prefers right-click+drag to select stations
  • Rectangle selection area on drag is unusable in specific circumstances, but using ctrl+click gives you enough granularity to deselect tasks when the rectangle selection area captures too many
• Finds that zoom-based clustering is OK, but would prefer sector or region-based clusters
• Sees value in both the expanded and collapsed task views - likes how the choice of expansion persists across task state
  • However, it seems that Pablo didn't notice that if you expand the task view, deselect the driver (so that nothing is selecting), and then re-select the driver, it defaults back to the collapsed task view.
• Based on experience with other softare, expected to be able to use a right-click+delete keybind to unassign tasks (that are selected)
• Thinks it's great that the corresponding station is highlighted when you hover over a task in the driver's task list
• Prefers assigning via the resource bar than the map
  • Likes being able to find a nearest driver if he's on one part of the map with a few tasks
• Likes being able to see full routes of driver (a bit too shaded out for him)
• Would love to see road details (ex. street names) more and at every zoom level
• Would like to see sectors because that's how they already operate (drivers are assigned sectors)
• Prefers circle-based clustering over heatmap design (because he can't tell where a cluster's borders start / end)
  • Believes this is a matter of preference, and so perhaps an option to toggle between views would be ideal

Key Takeaways

• Dragging as a primary mechanism for associating items to drivers isn't perfectly intuitive, but once the mechanism is learned, users naturally acquire more complex task-dispatching skills that build on the dragging mechanism.
• Whether drivers are assigned tasks via the map or the resource bar depends on factors such as whether the driver is in view on the map, whether there are other drivers overlapping (or near-overlapping) the target driver on the map, etc. Supporting both options is important.
• Grouping assigned tasks by station is preferred over the individual-tasks view (selected driver bar), but the distinction between these views isn't obvious enough.

Action Items & Issues Created

• Label collapsed task view as itinerary (to clarify distinction between task views) #982
• Allow re-assignment of task grouping #983
• Block re-ordering of task grouping when servicing at that station #984
• Block the Start Simulation button if no scenario is selected #975
• Block re-ordering of single task above servicing task in expanded view #989