Usability Tests

Table of Contents

• Usability Testing Plan
• Usability Testing Results - Round 1

---

Usability Testing Plan

The test plan addresses the how, when, where, who, why, and what of our usability testing. We've documented our approach to set a consistent framework that will serve us not only in this initial round of testing but in all subsequent rounds. Using the same test plan structure across multiple rounds allows us to:

• Maintain methodological consistency, making results comparable over time
• Onboard new team members or stakeholders more efficiently
• Track how user needs and pain points evolve as the product develops
• Build institutional knowledge about what works (and what doesn't) in our testing approach

The following sections detail the full usability testing program for VeloSim. On this Wiki page you can also find results from past rounds of testing.

Current goals and objectives

• Evaluate whether the current features meet the needs of all users
  • particularly BIXI dispatchers (the domain experts) as the primary target group
• Identify gaps between what end users need and what is currently provided
• Discover usability issues and pain points in existing features
• Prioritize future requirements based on user feedback and observed behaviour

Current research questions

• Current System Perception:
  • What are users' overall attitudes/satisfaction with the current system?
• Feature Assessment:
  • What tasks are users trying to accomplish that the current software doesn't support?
  • What features do users expect to find but couldn't?
  • Which existing features are intuitive vs. confusing or difficult to use?
• Workflow & Efficiency:
  • Where do users get stuck or frustrated in their workflows?
  • What would make users more efficient or productive in their work?
  • What workarounds, if any, are users creating to compensate for missing functionality?
  • How well does the software align with users' mental models of the problem domain?
• Unexpected behaviour in a production system:
  • Did end users encounter any issues, bugs, or unexpected behaviour?

Participant characteristics

These personas trace back to the types of users that we've documented on the Personas Wiki page.

Participant/Persona Type	Experience Level	Notes
Dispatcher	Domain experts and friends/family	Domain experts when available; friends/family debriefed with a script
Scenario Administrator	Simulated by friends and family	Debriefed with a script

Test method and setup

Test Design:

• Type: Moderated usability testing (formative evaluation)
• Estimated duration per session: 30 minutes (actual duration as of round 1 usability testing: 20 - 35 minutes)
• Structure: Think-aloud protocol with task completion

Environment:

• Platform: Production website (https://velosim.app)
• Devices: Desktop and laptop computers
  • While VeloSim's interface is responsive and supports mobile and tablet devices, the primary user base accesses the product on desktop and laptop computers. Accordingly, usability testing will focus on these platforms.

Data Collection:

• Screen capture and audio: OBS Studio
• Additional feedback: Post-test form via Google Forms

Moderator Protocol

1. Open the testing script in a browser window
2. Open VeloSim production environment (https://velosim.app)
3. Create a new non-admin user account for the participant (record credentials)
4. For Dispatcher personas: Download the test scenario file
5. Open the feedback form
6. Begin screen recording with OBS Studio
7. Hand computer to participant and instruct them to read the introduction and work through tasks
8. Observe and note any criticisms or issues during the session
9. Ensure participants complete checkpoint questions in the feedback form
10. After completion, stop recording, name video file with participant username, and upload to designated Google Drive folder

Participant task lists

Dispatcher Persona Tasks

Context: Participants role-play as BIXI task dispatchers who must manage battery swap operations. They assign tasks to drivers, prioritize urgent work, and monitor progress in a simulation environment.

Tasks:

1. Log into VeloSim using provided credentials
2. Import a pre-configured simulation scenario
3. Start the simulation
4. Zoom in to view station icons
5. Speed up the simulation
6. Wait for tasks to appear at stations
7. Pause the simulation
8. Click on stations to check existing tasks
9. Assign a task to a driver
10. Resume the simulation
11. Assign remaining tasks to the other driver
12. Reorder the task queue for a driver with multiple tasks
13. Explore additional features: unassigning tasks, dragging tasks to drivers

Checkpoints:

• After Tasks 1-2: Rate ease of login and scenario loading
• After Task 12: Rate ease of time navigation, task assignment, and task reordering
• Final: Complete general feedback section

Scenario Administrator Persona Tasks

Context: Participants create simulation scenarios by defining stations, drivers, vehicles, and scheduled tasks using the Scenario Editor.

Tasks:

1. Log into VeloSim using provided credentials
2. Navigate to Scenario Editor and create a new scenario
3. Draft a scenario following a detailed specification including:
   • Three stations with specific locations, initial tasks, and scheduled tasks
   • Two drivers with shift times and lunch breaks
   • Two vehicles with starting locations and battery inventory
4. Explore additional features: saving scenarios, creating duplicates, exporting to local computer, editing scenario titles, creating new versions vs. overriding existing scenarios

Checkpoints:

• After Task 3: Rate ease and effectiveness of Scenario Editor, suggest improvements, evaluate error message clarity
• Final: Complete general feedback section

Data to be collected and evaluation measures

• Quantitative data
  • Likert-scale questions about ease of use for feature set
  • Task-specific ease ratings (login, scenario loading, time navigation, task assignment, task re-ordering, scenario editor usage, error message clarity)
• Qualitative data
  • Observed pain points, confusions, and delights
  • Feature requests and suggestions
  • Positive feedback and what's working well
  • Open-ended feedback on overall system experience and suggested improvements

All quantitative and qualitative feedback will be collected through a standardized post-test Google form.

Report contents and presentation

• GitHub wiki page containing all our Usability Testing information
  • Summarizes the background of the study
    • including the goals, methodology, and participant characteristics
  • Present findings for original questions to investigate
  • Give user feedback results and discuss specifics (appropriate to data)
  • Suggest follow-up research
• SOEN 490 Release Presentation videos
  • Summarize usability testing plan and our results
  • Discuss how feedback was incorporated into design and implementation

Usability Testing Results - Round 1

We have maintained an ongoing collaborative feedback loop with our stakeholder throughout development, incorporating their input at multiple touchpoints including design mock-ups, live production testing, and backlog planning. Reaching our second release milestone provided an opportunity to conduct our very first round of structured usability testing. From January 13 to January 17, we engaged friends and family as test users, guiding them through a standardized bilingual script that evaluated core workflows of real BIXI dispatchers. We also had the opportunity to get feedback from 1 domain expert at BIXI (part of the core user base following the end of Release 3).

We collected both quantitative usability metrics and qualitative feedback to meet our initial objectives and answer the research questions described below.

Summary

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level	Notes
Dispatcher	12 (minimum)	22	Domain experts and friends/family	1 domain expert, 21 friends/family debriefed with a script
Scenario Administrator	3 (minimum)	9	Simulated by friends and family	Debriefed with a script

Note: Only 1 domain expert was available to participate during the Release 2 testing window.

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

Dispatcher Task Completion Time Results

TODO

Scenario Admin Survey Results

Likert Scale Questions (usability of scenario editor, helpfulness of error messages):

Scenario Admin Task Completion Time Results

TODO

General User Feedback Compilation

Below, feedback is split by type / source / feature group. Feedback from the domain expert is placed in its own category, since this feedback takes higher priority than others'.

Feedback that came from a domain expert

• The task assignment confirmation is important to have, but becomes annoying if you can't mass-assign tasks
• Multiple tasks should be selectable via both drag-to-select and control+click to select
• The vehicle icons should have driver names to distinguish them
  • In the real BIXI network, there are multiple task types - so we would also want the vehicle name displayed for each resource since different vehicles perform different tasks
  • Vehicle names would be displayed over the vehicle icon and in the resource bar
• The vehicle icon should be dynamic to display information related to resource state and direction of travel
• The selected item state should stay on station after a task assignment
  • This is mostly because, in the real BIXI netowrk, there are multiple task types
  • In the context of VeloSim, if mass-assignment is implemented, there is less concern about switching state on assignment
• Hard to see what stations are his priority, believes clusters would lead to a clean map but wouldn't help him find his priorities
• There should be the option to drag scenario files into the scenario editor to import them
• User isn't confident about being able to locate task priorities through the map alone
  • Hard to find stations with many tasks when looking for larger numbers in a sea of numbers
  • Believes a universal tasks bar would help in this regard
  • In the context of VeloSim (only battery-swap tasks), user doesn't care about stations with 0 tasks
  • In the real life BIXI network, where bike allocation tasks exist, user would care about all stations
• User believes it would be nice to be able to assign without dispatching (out of scope for our project as decided by stakeholder)

Feedback on the simulation page

• The icons, especially that of HQ, are confusing (8)
  • One proposed an icon legend
  • One proposed the ability to click HQ
• The process of assigning a task is difficult to learn (6)
  • One proposed putting a note in the driver's selected item bar
  • One proposed including a prompt to explain the task-assignment feature
  • One proposed highlighting the cars on the map and driver names in resource bar when a task enters drag mode
  • There is no affordance for dragging tasks, and no indication that you can drag them to the resource bar
• The automatic switch from station to vehicle as the selected item on task assignment is annoying (5)
• There should be a way to select and mass-assign tasks (4)
• The vehicle icons on the map should display the driver names or vehicle id above them (3)
• The task re-ordering UI should have friendlier bounds (3)
• There is nearly no visible disinction between 2 drivers in the same location (same station) due overlap (3)
  • It is hard to select assign tasks to overlapping vehicles
  • One proposition is to enlarge the car icon being hovered over
• There should be a way to select tasks in the same way as stations and vehicles (2)
  • There is no way to track that a task is assigned to, from the selected stations bar
• It is difficult to select a station when a driver is at that station due to icon overlap (2)
• There shouldn't be a confirmation dialog for task assignments (2)
• The meaning of the numbers on the HQ widget isn't clear (2)
• Confusion related to not being able to re-assign / re-order in-progress tasks (2)
• The selected item bar can hide too much of the map (2)
  • The task box is unecessarily long
  • One proposition is to make selected item bar translucent / invisible on task drag state
• There should be a built-in tutorial or documentation to reference on the site (2)
• The dots representing stations when zoomed out should be bigger (2)
• There should on-icon-hover-enabled tooltips for quick station / driver info (2)
• The sidebar on the simulation page should be removed or automatically minimized due to poor use of space (2)
• The large vehicle icon overlaps station dots when zoomed out far enough
• The simulation should be more realistic (traffic, weather events, construction, accidents)
• The car icon should reflect a bird's eye view of a vehicle
• The status of drivers should be included in the resource bar
• The selection of an entity should center the map on that entity
• The in-progress state should be split into two distinct states (on route and servicing)
• The process of re-ordering tasks is difficult to learn
• There is no easy way to tell what driver a task is assigned to
  • Could put a badge / label on the task item in the station's selected item card
  • Universal task sidebar that could include details about which driver has been assigned that task
• Desire to easily see how many tasks require assignment at each station, rather than (only) how many tasks total
• Desire to filter resource bar by driver name instead of just vehicle id
• Unsure what the number above the station represented
• Desire for universal tasks bar
• Want some way to know a driver is taking their break
• Would like marker clustering
  • Each cluster displays a number indicating how many stations it contains
• Would like sim-resumability
• Thought the arrow keys would speed up the sim
• Would like to know what the route would be before task assignment
• Needs to have more visual queues to understand what the simulation is doing

Unless stated otherwise, every point above was only mentioned once.

Feedback on the scenario editor page

- The use of the JSON format for defining scenarios is problematic (10) - Poor clarity, poor readability, difficulty troubleshooting validation issues, too text-heavy - Proposed alternatives include "something more visual" (map-based), or something form-based - One alternative is a helper modal to interact with the JSON (button to add driver, vehicle, station) - One suggestion was to offer support for a different format than JSON - The scenario editor lacks functionality expected in modern text editors (2) - Examples include tab-friendliness, line numbers, syntax coloring, and collapsable fields - The scenario editor should allow imports by dragging files in (2) - The scenario should save automatically when pressing *Start simulation* - The fonts used for field inputs differ from the fonts used for field labels - The font for the description field input shouldn't look like code (mono) - The JSON scenario fields should be collapsible - The import and export buttons should be nearer to one another - The save and export buttons should be the same colour - the import and new buttons should be the same colour - The scenario should be immediately editable upon selection - The scenario validation error messages are not helpful enough - There should be a way to make the scenario description field textarea bigger - There should be a dedicated *Validate scenario* button - The use of red for the selected scenario is confusing since red usually indicates something bad - The placement of the scenario description field before the scenario content field is unnatural

Unless stated otherwise, every point above was only mentioned once.

Feedback that was positive

Most feedback was critical of the application, but a few users highlighted things they liked about VeloSim:

• The flexibilty in how you can assign tasks (drag to icon or resource bar row) accomodates different needs (2)
• The ability to click on stations / vehicles to see more information about them is helpful
• The visualization of a driver's route is helpful
• The confirmation dialog that appears on attemtped task assignment prevents mistakes when dragging over vehicles that are near each other / overlapping
• The New button template in the scenario editor serves as a helpful guide

Unless stated otherwise, every point above was only mentioned once.

Feedback on the login page

• The login page should allow and ignore white-space around the username

Every point above was only mentioned once.

Feedback on admin-only users page

Some users were accidentally given an admin account to perform usability testing. The following is feedback they shared on the users dashboard.

• The Previous and Next buttons should remain at a fixed position when paginating
• The users table should be sortable by specified column
• The Id header of the users table should be labelled ID
• The users table should be filterable
• Admins should be able to see all currently running simulations, and be able to stop/pause them
• The ability to reset your own password from the sidebar should be more obvious
• The reset-password dialog should omit the obvious indication to "Click update when you're done"
• The create-use card should disappear after successful user creation

Every point above was only mentioned once.

Feedback that conflicts with project requirements

Some feedback either contradicts how the system is intended to work and can't be addressed without conflicting with project requirements.

• The numbering of tasks (their id generation) should be on a per-station level, not global
• There should be a built-in algorithm to automatically assign tasks to drivers already at a station
• The task lists belonging to stations should be re-orderable
• The tasks at a station should have a service priority

Every point above was only mentioned once.

Feedback on the usability test itself

Some feedback pertained to how the usability test was administered:

• Some didn't know what a "resources bar" was supposed to be (2)
• The test instructions should be clearer / simpler (2)
• The survey should not have a second open-ended question for scenario admins
• The usability test should introduce the notion of HQ in the beginning
• The stated user goal of ensuring efficient task assingments that save time and fuel is not satisfiable given the simplicity of the simulation used
• The dispatcher test should specify that the simulation is expected to have two stations
• The test instructions should not be in a separate tab
• The test should communicate the fact that you must zoom in to see the full station icons
• The tables in the scenario admin test should have titles that match the scenario attribute field names
• The system should be demoed to the user prior to attempting the usability test

Unless stated otherwise, every point above was only mentioned once.

Bugs discovered during usability testing

- Task assignment via dragging to the resources bar would highlight multiple driver rows instead of just one - A one-time 502 error was encountered in Firefox - Clicking on the current scenario after it has been edited shows a "Unsaved Changes" card - User search bar in the users dashboard does not work when looking at a non-first page

Unless stated otherwise, every point above was only mentioned once.

Key Takeaways

• The simulation page needs its own mechanisms to explain its various parts to users (e.g. a legend for icons, task assignment mechanics, etc.)
• The simulation page needs more affordances for possible actions and clearer feedback about task-driver relationships
• The simulation page needs to provide more options for flexible and efficient task assignment
• Vehicle icons on the map need more identifying characteristics (i.e. driver name, vehicle ID, direction of route)
• Entity icons on the map generally suffer from overlap-induced visibility issues
• The scenario editor page needs to have more options for non-technical users
• The scenario content text area needs to be more feature rich due to the size of content being worked with there
• The scenario editor page should allow imports via dragging since this design pattern is commonplace

Action Items & Issues Created

Simulation Page Improvements

• Enable mass-assignment of tasks via click selection
• Enable mass-assignment of tasks via drag selection
• Add driver names above vehicle icons
• Add resource state information and direction of travel to vehicle icons
• Add resource state information to resource bar
• Add resource state information to selected item bar

• Maintain selected item state on task assignment

• Reduce visibility of 0-task stations

Scenario Editor Page Improvements

• Offer per-entity forms for scenario manipulation
• Add line numbers to the scenario content textarea
• Add drag and drop support for scenario importing #456

Bug Fixes

• Resource bar task-assignment hover persists
• Unsaved changes card appears when attempting navigating to current scenario
• Users search bar breaks past first page of users

Usability Test Improvements

• Explain what the resource bar is supposed to be in the dispatcher usability test script
• Introduce the notion of HQ in the introduction portion of each usability test script
• Remove open-ended question from scenario admin section of survey to prevent response duplication
• Remove the stated user goal of ensuring efficient task assingments that "save time and fuel" (not really applicable to the scenario used for usability testing)
• Add information pertaining to the total number of stations in the dispatcher usabilty test script
• Update the table names in the scenario admin test to match the scenario attribute field names