Usability Tests

Table of Contents

• Usability Testing Plan
• Usability Testing Results - Round 1

---

Usability Testing Plan

The test plan addresses the how, when, where, who, why, and what of our usability testing. We've documented our approach to set a consistent framework that will serve us not only in this initial round of testing but in all subsequent rounds. Using the same test plan structure across multiple rounds allows us to:

• Maintain methodological consistency, making results comparable over time
• Onboard new team members or stakeholders more efficiently
• Track how user needs and pain points evolve as the product develops
• Build institutional knowledge about what works (and what doesn't) in our testing approach

The following sections detail the full usability testing program for VeloSim. On this Wiki page you can also find results from past rounds of testing.

Current goals and objectives

• Evaluate whether the current features meet the needs of all users
  • particularly BIXI dispatchers (the domain experts) as the primary target group
• Identify gaps between what end users need and what is currently provided
• Discover usability issues and pain points in existing features
• Prioritize future requirements based on user feedback and observed behaviour

Current research questions

• Current System Perception:
  • What are users' overall attitudes/satisfaction with the current system?
• Feature Assessment:
  • What tasks are users trying to accomplish that the current software doesn't support?
  • What features do users expect to find but couldn't?
  • Which existing features are intuitive vs. confusing or difficult to use?
• Workflow & Efficiency:
  • Where do users get stuck or frustrated in their workflows?
  • What would make users more efficient or productive in their work?
  • What workarounds, if any, are users creating to compensate for missing functionality?
  • How well does the software align with users' mental models of the problem domain?
• Unexpected behaviour in a production system:
  • Did end users encounter any issues, bugs, or unexpected behaviour?

Participant characteristics

These personas trace back to the types of users that we've documented on the Personas Wiki page.

Participant/Persona Type	Experience Level	Notes
Dispatcher	Domain experts and friends/family	Domain experts when available; friends/family debriefed with a script
Scenario Administrator	Simulated by friends and family	Debriefed with a script

Test method and setup

Test Design:

• Type: Moderated usability testing (formative evaluation)
• Estimated duration per session: 30 minutes (actual duration as of round 1 usability testing: 20 - 35 minutes)
• Structure: Think-aloud protocol with task completion

Environment:

• Platform: Production website (https://velosim.app)
• Devices: Desktop and laptop computers
  • While VeloSim's interface is responsive and supports mobile and tablet devices, the primary user base accesses the product on desktop and laptop computers. Accordingly, usability testing will focus on these platforms.

Data Collection:

• Screen capture and audio: OBS Studio
• Additional feedback: Post-test form via Google Forms

Moderator Protocol

1. Open the testing script in a browser window
2. Open VeloSim production environment (https://velosim.app)
3. Create a new non-admin user account for the participant (record credentials)
4. For Dispatcher personas: Download the test scenario file
5. Open the feedback form
6. Begin screen recording with OBS Studio
7. Hand computer to participant and instruct them to read the introduction and work through tasks
8. Observe and note any criticisms or issues during the session
9. Ensure participants complete checkpoint questions in the feedback form
10. After completion, stop recording, name video file with participant username, and upload to designated Google Drive folder

Participant task lists

Dispatcher Persona Tasks

Context: Participants role-play as BIXI task dispatchers who must manage battery swap operations. They assign tasks to drivers, prioritize urgent work, and monitor progress in a simulation environment.

Tasks:

1. Log into VeloSim using provided credentials
2. Import a pre-configured simulation scenario
3. Start the simulation
4. Zoom in to view station icons
5. Speed up the simulation
6. Wait for tasks to appear at stations
7. Pause the simulation
8. Click on stations to check existing tasks
9. Assign a task to a driver
10. Resume the simulation
11. Assign remaining tasks to the other driver
12. Reorder the task queue for a driver with multiple tasks
13. Explore additional features: unassigning tasks, dragging tasks to drivers

Checkpoints:

• After Tasks 1-2: Rate ease of login and scenario loading
• After Task 12: Rate ease of time navigation, task assignment, and task reordering
• Final: Complete general feedback section

Scenario Administrator Persona Tasks

Context: Participants create simulation scenarios by defining stations, drivers, vehicles, and scheduled tasks using the Scenario Editor.

Tasks:

1. Log into VeloSim using provided credentials
2. Navigate to Scenario Editor and create a new scenario
3. Draft a scenario following a detailed specification including:
   • Three stations with specific locations, initial tasks, and scheduled tasks
   • Two drivers with shift times and lunch breaks
   • Two vehicles with starting locations and battery inventory
4. Explore additional features: saving scenarios, creating duplicates, exporting to local computer, editing scenario titles, creating new versions vs. overriding existing scenarios

Checkpoints:

• After Task 3: Rate ease and effectiveness of Scenario Editor, suggest improvements, evaluate error message clarity
• Final: Complete general feedback section

Data to be collected and evaluation measures

• Quantitative data
  • Likert-scale questions about ease of use for feature set
  • System Usability Scale (SUS) questions
  • Task-specific ease ratings (login, scenario loading, time navigation, task assignment, task re-ordering, scenario editor usage, error message clarity)
• Qualitative data
  • Observed pain points, confusions, and delights
  • Feature requests and suggestions
  • Positive feedback and what's working well
  • Open-ended feedback on overall system experience and suggested improvements

All quantitative and qualitative feedback will be collected through a standardized post-test Google form.

Report contents and presentation

• GitHub wiki page containing all our Usability Testing information
  • Summarizes the background of the study
    • including the goals, methodology, and participant characteristics
  • Present findings for original questions to investigate
  • Give user feedback results and discuss specifics (appropriate to data)
  • Suggest follow-up research
• SOEN 490 Release Presentation videos
  • Summarize usability testing plan and our results
  • Discuss how feedback was incorporated into design and implementation

Usability Testing Results - Round 1

We have maintained an ongoing collaborative feedback loop with our stakeholder throughout development, incorporating their input at multiple touchpoints including design mock-ups, live production testing, and backlog planning. Reaching our second release milestone provided an opportunity to conduct our very first round of structured usability testing. From January 13 to January 17, we engaged friends and family as test users, guiding them through a standardized bilingual script that evaluated core workflows of real BIXI dispatchers. We also had the opportunity to get feedback from 1 domain expert at BIXI (part of the core user base following the end of Release 3).

We collected both quantitative usability metrics and qualitative feedback to meet our initial objectives and answer the research questions described below.

Summary

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level	Notes
Dispatcher	12 (minimum)	22	Domain experts and friends/family	1 domain expert, 21 friends/family debriefed with a script
Scenario Administrator	3 (minimum)	9	Simulated by friends and family	Debriefed with a script

Note: Only 1 domain expert was available to participate during the Release 2 testing window.

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

Dispatcher Task Completion Time Results

TODO

Scenario Admin Survey Results

Likert Scale Questions (usability of scenario editor, helpfulness of error messages):

Open-ended Questions (scenario editor feedback):

Feedback pertaining to the scenario editor:

• Issues due to the use of the JSON format to define the scenarios (clarity, readability, troubleshooting validation issues, too text-heavy) (7 mentions)
  • Alternatives proposed include a different format than JSON, form-based (4), a datetime picker for timestamp fields, map-based (world-builder-esque)
• The scenario editor lacks functionality expected in text editors (tab-friendliness, line numbers, syntax coloring, collapsable fields) (2 mention)
• Difficult to understand what to delete and what not to delete in the json editor (1 mention)
• The scenario doesn't save automatically when pressing Start simulation (1 mention)
• Desire for the ability to drag a JSON file over the editor to import / save it (1 mention)
• The use of red for the selected scenario is confusing since red is usually used to indicate something bad (1 mention)
• The placement of the scenario description field before the scenario content field is unatural (1 mention)

Feedback pertaining to the login page:

• Log in page should allow white-space around the username (1 mention)

Feedback pertaining to the usability test:

• Titles in the task tables should match the json titles (1 mention)
• Instructions could be a little simplified (1 mention)

Bugs discovered:

• Clicking on the current scenario after it has been edited shows a "Unsaved Changes" card (1 mention)

Scenario Admin Task Completion Time Results

TODO

General User Feedback Compilation

Simulation Page Negative:

• Icon confusion / desire for an icon legend (esp. HQ) (8 mentions)
  • Would like ability to click HQ (1)
• Expected ability to "select" tasks in the same way as stations / vehicles (2 mentions)
  • How else to know how to assign the task / how to track the driver that would have been assigned a task?
  • Display task type as information
• Large vehicle icon overlaps station dots when zoomed out far enough (1 mention)
• Hard to select a station when a driver is at that station (icon overlap) (2 mentions)
• 2 drivers in the same location (same station) overlap and look like 1 icon (1 mention)
• Annoyance related to having to confirm task assignments (2 mentions)
• Desire for ability to select and mass-assign tasks (4 mentions)
• Difficulty understanding how to assign a task (6 mentions)
  • Could put a note in the driver's selected item bar (1)
  • Could put a prompt to explain feature (1)
  • Could highlight the cars / driver names in resource bar when a task enters drag mode (1)
  • No affordance for dragging tasks (also don't know you can drag to the resource bar)
• Desire for the sim to be more realistic (traffic, weather events, construction, accidents) (1 mention)
• Would be nice to have driver names / ids above the vehicle icons (3 mentions)
• Driver HQ widget confusion (not sure what the numbers mean) (2 mentions)
• Can't easily select the vehicle you want when 2 are in the same position (without resorting to resource bar) (1 mention)
• Unsure which vehicle a task is being assigned to (1 mention)
  • perhaps enlarge the car icon being hovered over (1)
• car icon could be changed to reflect a bird's eye view of it (1 mention)
• Switch from station to vehicle as selected item on task assignment annoyance (5 mentions)
• use hover tooltips for quick station/driver info so that I don't have to click and view the UI info section each time (2 mention)
• include a status for drivers on resource bar ex. 'driving back to HQ', 'servicing task X'
• centering the map on the selected entity upon selection of that entity (1 mention)
• desire to split in progress state into two distinct states (1 mention)
• Confusion related to not being able to re-assign / re-order in-progress tasks (2 mentions)
• improve sliding/bounds for task reordering UI items (3 mentions)
• Not sure how to re-order (1)
• Hard to tell what driver a task is assigned to (1 mention)
  • Could put a badge / label on the task item in the station's selected item card (1)
  • universal task sidebar that could include details about which driver has been assigned that task (1)
• Desire to easily see how many tasks require assignment at each station, rather than (only) how many tasks total (1)
• Desire to filter resource bar by driver name instead of just vehicle id.
• selected item bar can hide too much of the map (2 mentions)
  • Task box is unecessarily long (1)
  • Perhaps make selected item bar translucent / invisible on task drag state (1)
• Didn't know what a "resources bar" was supposed to be. (2 mentions)
• Desire for built-in tutorial or documentation to reference on the site (2 mentions)
• when you're zoomed out and there's a dot, and there's a task, make it obvious that there's a task (1 mention)
  • maybe
• unsure what the number above the station represented (1 mention)
• desire for universal tasks bar
• Want some way to know a driver is taking their break (1 mention)
• Would like marker clustering (1 mention)
  • Each cluster displays a number indicating how many stations it contains (wouldn't total task count be better?) (1 mention)
• Would like bigger station dots when zoomed out (2 mentions)
• would like sim-resumability
• thought the arrow keys would speed up the sim (1 mention)
• would like to know what the route would be before task assignment (1 mention)
• Thinks sidebar should be minized by default when a simulation starts (1 mention)
• needs to have more visual queues to understand what the simulation is doing (1 mention)

Simulation Page Positive:

• Being able to click on entities to see more information (stations, vehicles) (1 mention)
• Likes the route visualization (1 mention)
• Liked that a confirmation appears to show which driver a task is being assigned to to avoid conflicts (1 mention)
• Likes that there's multiple ways to assign a task (icon + resource bar row) (2 mentions)

Scenario editor page negative:

• Distaste for use of JSON format for scenario content (too technical) (8 mentions)
  • Alternatives include "something more visual" (2), form-based (2)
  • helper modal to interact with the JSON (button to add driver, vehicle, station) (1)
• Font for the Scenario name and description/JSON fields are different (1 mention)
• Description box's font to be something non-code related (1 mention)
• Desire for collapsible JSON fields (1 mention)
• Expected Import and Export buttons to be together, and are distant (1 mention)
• Expected to be able to immediately able to edit a scenario (without needing to press the edit button) (1 mention)
  • "I would imagine right clicking a scenario item, pressing edit, and then the contents showing in the editor with all buttons pressable"
• Believed they could import scenario files by dragging them into the scenario editor page (1 mention)
• Error messages not helpful enough (1 mention)
• Have a way to grow the scenario description text area (1 mention)
• Desire for a non-save, separate validate button (1 mention)
• doesn't like save / export buttons being different colours (1 mention)
• doesn't like import / new buttons being separate colours (1 mention)

scenario editor page positive:

• liked that if you press 'new' on the scenario, you get a template to guide you!

Users Page Negative:

• "Previous / Next" should remain at a fixed position when navigating (1 mention)

• Desire for user sorting capabilites (1 mention)

• Preference for 'Id' column to be labelled 'ID' (1 mention)

• Desire for filters on the users table (1 mention)

• Desire for admins to see current running simulations, and be able to stop/pause them (1 mention)

• Can't tell that you can reset your password by clicking on the user icon in the sidebar (1 mention)

• "Click update when you're done." should be omitted from update password card as its obvious (1 mention)

• Expected task numbering to be per-station and not global (1 mention)

• Desire for built-in algorithm to automatically assign+service tasks to drivers already at a station (1 mention)

• Desire to be able to re-order station tasks (1 mention)

• asked about adding a priority for tasks to be serviced at a station (1 mention)

• sidebar on the sim page feels unnecessary if we're not going to have lots of tabs and could be replace with something simpler that does not add as much white/empty space (1 mention)

• User search bar does not work when looking at a non-first page. I believe it is a bug. (1 mention)

• Expected create user card to disappear after successfully creating a new user (1 mention)

• "Have someone show me the system before and how everything works all together." (1 mention)

Bugs found:

• visual bug for drag-drop task assignment highlighting multiple driver item bars instead of just 1 (see recording) (1 mention)
• Encountered a one-time 502 error in Firefox; issue did not reproduce in Chrome (may have been caused by a browser-specific or client-side condition) (1 mention)

Usability test feedback:

• "I already provided it on the last page. this is a problem to ask me this question again." (1 mention)
• Had no idea about the notion of HQ even after the introduction to the usability test (1 mention)
• didn't feel like ensuring task allocation was "efficient to save time and fuel" was an actual part of the test (1 mention)
• Test should mention that we're looking for two stations - since icons are hard to see.
• "Clearer instructions, explain as if i was a 3 year old."
• "Instructions should be directly on the screen." (1 mention)
• didn't know how close to zoom in to make the station icons appear since she didn't know they exist (1 mention)

Key Takeaways

TODO

Action Items & Issues Created

TODO