Usability Tests

Table of Contents

• Usability Testing Plan
• Usability Testing Results - Round 1

---

Usability Testing Plan

The test plan addresses the how, when, where, who, why, and what of our usability testing. We've documented our approach to set a consistent framework that will serve us not only in this initial round of testing but in all subsequent rounds. Using the same test plan structure across multiple rounds allows us to:

• Maintain methodological consistency, making results comparable over time
• Onboard new team members or stakeholders more efficiently
• Track how user needs and pain points evolve as the product develops
• Build institutional knowledge about what works (and what doesn't) in our testing approach

The following sections detail the full usability testing program for VeloSim. On this Wiki page you can also find results from past rounds of testing.

Current goals and objectives

• Evaluate whether the current features meet the needs of all users
  • particularly BIXI dispatchers (the domain experts) as the primary target group
• Identify gaps between what end users need and what is currently provided
• Discover usability issues and pain points in existing features
• Prioritize future requirements based on user feedback and observed behaviour

Current research questions

• Current System Perception:
  • What are users' overall attitudes/satisfaction with the current system?
• Feature Assessment:
  • What tasks are users trying to accomplish that the current software doesn't support?
  • What features do users expect to find but couldn't?
  • Which existing features are intuitive vs. confusing or difficult to use?
• Workflow & Efficiency:
  • Where do users get stuck or frustrated in their workflows?
  • What would make users more efficient or productive in their work?
  • What workarounds, if any, are users creating to compensate for missing functionality?
  • How well does the software align with users' mental models of the problem domain?
• Unexpected behaviour in a production system:
  • Did end users encounter any issues, bugs, or unexpected behaviour?

Participant characteristics

These personas trace back to the types of users that we've documented on the Personas Wiki page.

Participant/Persona Type	Experience Level	Notes
Dispatcher	Domain experts and friends/family	Domain experts when available; friends/family debriefed with a script
Scenario Administrator	Simulated by friends and family	Debriefed with a script

Test method and setup

Test Design:

• Type: Moderated usability testing (formative evaluation)
• Estimated duration per session: 30 minutes (actual duration as of round 1 usability testing: 20 - 35 minutes)
• Structure: Think-aloud protocol with task completion

Environment:

• Platform: Production website (https://velosim.app)
• Devices: Desktop and laptop computers
  • While VeloSim's interface is responsive and supports mobile and tablet devices, the primary user base accesses the product on desktop and laptop computers. Accordingly, usability testing will focus on these platforms.

Data Collection:

• Screen capture and audio: OBS Studio
• Additional feedback: Post-test form via Google Forms

Moderator Protocol

1. Open the testing script in a browser window
2. Open VeloSim production environment (https://velosim.app)
3. Create a new non-admin user account for the participant (record credentials)
4. For Dispatcher personas: Download the test scenario file
5. Open the feedback form
6. Begin screen recording with OBS Studio
7. Hand computer to participant and instruct them to read the introduction and work through tasks
8. Observe and note any criticisms or issues during the session
9. Ensure participants complete checkpoint questions in the feedback form
10. After completion, stop recording, name video file with participant username, and upload to designated Google Drive folder

Participant task lists

Dispatcher Persona Tasks

Context: Participants role-play as BIXI task dispatchers who must manage battery swap operations. They assign tasks to drivers, prioritize urgent work, and monitor progress in a simulation environment.

Tasks:

1. Log into VeloSim using provided credentials
2. Import a pre-configured simulation scenario
3. Start the simulation
4. Zoom in to view station icons
5. Speed up the simulation
6. Wait for tasks to appear at stations
7. Pause the simulation
8. Click on stations to check existing tasks
9. Assign a task to a driver
10. Resume the simulation
11. Assign remaining tasks to the other driver
12. Reorder the task queue for a driver with multiple tasks
13. Explore additional features: unassigning tasks, dragging tasks to drivers

Checkpoints:

• After Tasks 1-2: Rate ease of login and scenario loading
• After Task 12: Rate ease of time navigation, task assignment, and task reordering
• Final: Complete general feedback section

Scenario Administrator Persona Tasks

Context: Participants create simulation scenarios by defining stations, drivers, vehicles, and scheduled tasks using the Scenario Editor.

Tasks:

1. Log into VeloSim using provided credentials
2. Navigate to Scenario Editor and create a new scenario
3. Draft a scenario following a detailed specification including:
   • Three stations with specific locations, initial tasks, and scheduled tasks
   • Two drivers with shift times and lunch breaks
   • Two vehicles with starting locations and battery inventory
4. Explore additional features: saving scenarios, creating duplicates, exporting to local computer, editing scenario titles, creating new versions vs. overriding existing scenarios

Checkpoints:

• After Task 3: Rate ease and effectiveness of Scenario Editor, suggest improvements, evaluate error message clarity
• Final: Complete general feedback section

Data to be collected and evaluation measures

• Quantitative data
  • Likert-scale questions about ease of use for feature set
  • System Usability Scale (SUS) questions
  • Task-specific ease ratings (login, scenario loading, time navigation, task assignment, task re-ordering, scenario editor usage, error message clarity)
• Qualitative data
  • Observed pain points, confusions, and delights
  • Feature requests and suggestions
  • Positive feedback and what's working well
  • Open-ended feedback on overall system experience and suggested improvements

All quantitative and qualitative feedback will be collected through a standardized post-test Google form.

Report contents and presentation

• GitHub wiki page containing all our Usability Testing information
  • Summarizes the background of the study
    • including the goals, methodology, and participant characteristics
  • Present findings for original questions to investigate
  • Give user feedback results and discuss specifics (appropriate to data)
  • Suggest follow-up research
• SOEN 490 Release Presentation videos
  • Summarize usability testing plan and our results
  • Discuss how feedback was incorporated into design and implementation

Usability Testing Results - Round 1

We have maintained an ongoing collaborative feedback loop with our stakeholder throughout development, incorporating their input at multiple touchpoints including design mock-ups, live production testing, and backlog planning. Reaching our second release milestone provided an opportunity to conduct our very first round of structured usability testing. From January 13 to January 17, we engaged friends and family as test users, guiding them through a standardized bilingual script that evaluated core workflows of real BIXI dispatchers. We also had the opportunity to get feedback from 1 domain expert at BIXI (part of the core user base following the end of Release 3).

We collected both quantitative usability metrics and qualitative feedback to meet our initial objectives and answer the research questions described below.

Summary

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level	Notes
Dispatcher	12 (minimum)	22	Domain experts and friends/family	1 domain expert, 21 friends/family debriefed with a script
Scenario Administrator	3 (minimum)	9	Simulated by friends and family	Debriefed with a script

Note: Only 1 domain expert was available to participate during the Release 2 testing window.

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

Dispatcher Task Completion Time Results

TODO

Scenario Admin Survey Results

Likert Scale Questions (usability of scenario editor, helpfulness of error messages):

Scenario Admin Task Completion Time Results

TODO

General User Feedback Compilation

Simulation Page Negative:

• Icon confusion / desire for an icon legend (esp. HQ) (8 mentions)
  • Would like ability to click HQ (1)
• Difficulty understanding how to assign a task (6 mentions)
  • Could put a note in the driver's selected item bar (1)
  • Could put a prompt to explain feature (1)
  • Could highlight the cars / driver names in resource bar when a task enters drag mode (1)
  • No affordance for dragging tasks (also don't know you can drag to the resource bar)
• Switch from station to vehicle as selected item on task assignment annoyance (5 mentions)
• Desire for ability to select and mass-assign tasks (4 mentions)
• Would be nice to have driver names / ids above the vehicle icons (3 mentions)
• improve sliding/bounds for task reordering UI items (3 mentions)
• Expected ability to "select" tasks in the same way as stations / vehicles (2 mentions)
  • How else to know how to assign the task / how to track the driver that would have been assigned a task?
  • Display task type as information
• Hard to select a station when a driver is at that station (icon overlap) (2 mentions)
• Annoyance related to having to confirm task assignments (2 mentions)
• Driver HQ widget confusion (not sure what the numbers mean) (2 mentions)
• Confusion related to not being able to re-assign / re-order in-progress tasks (2 mentions)
• selected item bar can hide too much of the map (2 mentions)
  • Task box is unecessarily long (1)
  • Perhaps make selected item bar translucent / invisible on task drag state (1)
• Didn't know what a "resources bar" was supposed to be. (2 mentions)
• Desire for built-in tutorial or documentation to reference on the site (2 mentions)
• Would like bigger station dots when zoomed out (2 mentions)
• Large vehicle icon overlaps station dots when zoomed out far enough (1 mention)
• 2 drivers in the same location (same station) overlap and look like 1 icon (1 mention)
• Desire for the sim to be more realistic (traffic, weather events, construction, accidents) (1 mention)
• Can't easily select the vehicle you want when 2 are in the same position (without resorting to resource bar) (1 mention)
• Unsure which vehicle a task is being assigned to (1 mention)
  • perhaps enlarge the car icon being hovered over (1)
• car icon could be changed to reflect a bird's eye view of it (1 mention)
• use hover tooltips for quick station/driver info so that I don't have to click and view the UI info section each time (2 mention)
• include a status for drivers on resource bar ex. 'driving back to HQ', 'servicing task X'
• centering the map on the selected entity upon selection of that entity (1 mention)
• desire to split in progress state into two distinct states (1 mention)
• Not sure how to re-order (1)
• Hard to tell what driver a task is assigned to (1 mention)
  • Could put a badge / label on the task item in the station's selected item card (1)
  • universal task sidebar that could include details about which driver has been assigned that task (1)
• Desire to easily see how many tasks require assignment at each station, rather than (only) how many tasks total (1)
• Desire to filter resource bar by driver name instead of just vehicle id.
• when you're zoomed out and there's a dot, and there's a task, make it obvious that there's a task (1 mention)
  • maybe
• unsure what the number above the station represented (1 mention)
• desire for universal tasks bar
• Want some way to know a driver is taking their break (1 mention)
• Would like marker clustering (1 mention)
  • Each cluster displays a number indicating how many stations it contains (wouldn't total task count be better?) (1 mention)
• would like sim-resumability
• thought the arrow keys would speed up the sim (1 mention)
• would like to know what the route would be before task assignment (1 mention)
• Thinks sidebar should be minized by default when a simulation starts (1 mention)
• needs to have more visual queues to understand what the simulation is doing (1 mention)
• The sidebar on the simulation page should be removed due to poor use of space (1 mention)

Scenario editor page negative:

• Distaste for use of JSON format for scenario content (too technical) (10 mentions)
  • (clarity, readability, troubleshooting validation issues, too text-heavy)
  • Alternatives include "something more visual" (2), form-based (2)
  • helper modal to interact with the JSON (button to add driver, vehicle, station) (1)
  • map-based (world-builder-esque)
  • supporting a different format than JSON
• The scenario editor lacks functionality expected in text editors (tab-friendliness, line numbers, syntax coloring, collapsable fields) (2 mention)
• Believed they could import scenario files by dragging them into the scenario editor page (2 mention)
• Font for the Scenario name and description/JSON fields are different (1 mention)
• Description box's font to be something non-code related (1 mention)
• Desire for collapsible JSON fields (1 mention)
• Expected Import and Export buttons to be together, and are distant (1 mention)
• Expected to be able to immediately able to edit a scenario (without needing to press the edit button) (1 mention)
  • "I would imagine right clicking a scenario item, pressing edit, and then the contents showing in the editor with all buttons pressable"
• Error messages not helpful enough (1 mention)
• Have a way to grow the scenario description text area (1 mention)
• Desire for a non-save, separate validate button (1 mention)
• doesn't like save / export buttons being different colours (1 mention)
• doesn't like import / new buttons being separate colours (1 mention)
• The scenario doesn't save automatically when pressing Start simulation (1 mention)
• The use of red for the selected scenario is confusing since red is usually used to indicate something bad (1 mention)
• The placement of the scenario description field before the scenario content field is unatural (1 mention)

Feedback that was positive

Most feedback was critical of the application, but a few users highlighted things they liked about VeloSim:

• The flexibilty in how you can assign tasks (drag to icon or resource bar row) accomodates different needs (2)
• The ability to click on stations / vehicles to see more information about them is helpful
• The visualization of a driver's route is helpful
• The confirmation dialog that appears on attemtped task assignment prevents mistakes when dragging over vehicles that are near each other / overlapping
• The New button template in the scenario editor serves as a helpful guide

Feedback on the login page

- The login page should allow and ignore white-space around the username

Every point above was only mentioned once.

Feedback on admin-only users page

Some users were accidentally given an admin account to perform usability testing. The following is feedback they shared on the users dashboard.

• The Previous and Next buttons should remain at a fixed position when paginating
• The users table should be sortable by specified column
• The Id header of the users table should be labelled ID
• The users table should be filterable
• Admins should be able to see all currently running simulations, and be able to stop/pause them
• The ability to reset your own password from the sidebar should be more obvious
• The reset-password dialog should omit the obvious indication to "Click update when you're done"
• The create-use card should disappear after successful user creation

Every point above was only mentioned once.

Feedback that conflicts with project requirements

Some feedback either contradicts how the system is intended to work and can't be addressed without conflicting with project requirements.

• The numbering of tasks (their id generation) should be on a per-station level, not global
• There should be a built-in algorithm to automatically assign tasks to drivers already at a station
• The task lists belonging to stations should be re-orderable
• The tasks at a station should have a service priority

Every point above was only mentioned once.

Feedback on the usability test itself

Some feedback pertained to how the usability test was administered:

• The test instructions should be clearer / simpler (2)
• The survey should not have a second open-ended question for scenario admins
• The usability test should introduce the notion of HQ in the beginning
• The stated user goal of ensuring efficient task assingments that save time and fuel is not satisfiable given the simplicity of the simulation used
• The dispatcher test should specify that the simulation is expected to have two stations
• The test instructions should not be in a separate tab
• The test should communicate the fact that you must zoom in to see the full station icons
• The tables in the scenario admin test should have titles that match the scenario attribute field names
• The system should be demoed to the user prior to attempting the usability test

Unless stated otherwise, every point above was only mentioned once.

Bugs discovered during usability testing

- Task assignment via dragging to the resources bar would highlight multiple driver rows instead of just one - A one-time 502 error was encountered in Firefox - Clicking on the current scenario after it has been edited shows a "Unsaved Changes" card - User search bar in the users dashboard does not work when looking at a non-first page

Every point above was only mentioned once.

Key Takeaways

TODO

Action Items & Issues Created

TODO