Usability Tests

Table of Contents

• Usability Testing Plan
• Usability Testing Results - Round 1

---

Usability Testing Plan

The test plan addresses the how, when, where, who, why, and what of our usability testing. We've documented our approach to set a consistent framework that will serve us not only in this initial round of testing but in all subsequent rounds. Using the same test plan structure across multiple rounds allows us to:

• Maintain methodological consistency, making results comparable over time
• Onboard new team members or stakeholders more efficiently
• Track how user needs and pain points evolve as the product develops
• Build institutional knowledge about what works (and what doesn't) in our testing approach

The following sections detail the full usability testing program for VeloSim. On this Wiki page you can also find results from past rounds of testing.

Current goals and objectives

• Evaluate whether the current features meet the needs of all users
  • particularly BIXI dispatchers (the domain experts) as the primary target group
• Identify gaps between what end users need and what is currently provided
• Discover usability issues and pain points in existing features
• Prioritize future requirements based on user feedback and observed behaviour

Current research questions

• Current System Perception:
  • What are users' overall attitudes/satisfaction with the current system?
• Feature Assessment:
  • What tasks are users trying to accomplish that the current software doesn't support?
  • What features do users expect to find but couldn't?
  • Which existing features are intuitive vs. confusing or difficult to use?
• Workflow & Efficiency:
  • Where do users get stuck or frustrated in their workflows?
  • What would make users more efficient or productive in their work?
  • What workarounds, if any, are users creating to compensate for missing functionality?
  • How well does the software align with users' mental models of the problem domain?
• Unexpected behaviour in a production system:
  • Did end users encounter any issues, bugs, or unexpected behaviour?

Participant characteristics

These personas trace back to the types of users that we've documented on the Personas Wiki page.

Participant/Persona Type	Experience Level	Notes
Dispatcher	Domain experts and friends/family	Domain experts when available; friends/family debriefed with a script
Scenario Administrator	Simulated by friends and family	Debriefed with a script

Test method and setup

Test Design:

• Type: Moderated usability testing (formative evaluation)
• Estimated duration per session: 30 minutes (actual duration as of round 1 usability testing: 20 - 35 minutes)
• Structure: Think-aloud protocol with task completion

Environment:

• Platform: Production website (https://velosim.app)
• Devices: Desktop and laptop computers
  • While VeloSim's interface is responsive and supports mobile and tablet devices, the primary user base accesses the product on desktop and laptop computers. Accordingly, usability testing will focus on these platforms.

Data Collection:

• Screen capture and audio: OBS Studio
• Additional feedback: Post-test form via Google Forms

Moderator Protocol

1. Open the testing script in a browser window
2. Open VeloSim production environment (https://velosim.app)
3. Create a new non-admin user account for the participant (record credentials)
4. For Dispatcher personas: Download the test scenario file
5. Open the feedback form
6. Begin screen recording with OBS Studio
7. Hand computer to participant and instruct them to read the introduction and work through tasks
8. Observe and note any criticisms or issues during the session
9. Ensure participants complete checkpoint questions in the feedback form
10. After completion, stop recording, name video file with participant username, and upload to designated Google Drive folder

Participant task lists

Dispatcher Persona Tasks

Context: Participants role-play as BIXI task dispatchers who must manage battery swap operations. They assign tasks to drivers, prioritize urgent work, and monitor progress in a simulation environment.

Tasks:

1. Log into VeloSim using provided credentials
2. Import a pre-configured simulation scenario
3. Start the simulation
4. Zoom in to view station icons
5. Speed up the simulation
6. Wait for tasks to appear at stations
7. Pause the simulation
8. Click on stations to check existing tasks
9. Assign a task to a driver
10. Resume the simulation
11. Assign remaining tasks to the other driver
12. Reorder the task queue for a driver with multiple tasks
13. Explore additional features: unassigning tasks, dragging tasks to drivers

Checkpoints:

• After Tasks 1-2: Rate ease of login and scenario loading
• After Task 12: Rate ease of time navigation, task assignment, and task reordering
• Final: Complete general feedback section

Scenario Administrator Persona Tasks

Context: Participants create simulation scenarios by defining stations, drivers, vehicles, and scheduled tasks using the Scenario Editor.

Tasks:

1. Log into VeloSim using provided credentials
2. Navigate to Scenario Editor and create a new scenario
3. Draft a scenario following a detailed specification including:
   • Three stations with specific locations, initial tasks, and scheduled tasks
   • Two drivers with shift times and lunch breaks
   • Two vehicles with starting locations and battery inventory
4. Explore additional features: saving scenarios, creating duplicates, exporting to local computer, editing scenario titles, creating new versions vs. overriding existing scenarios

Checkpoints:

• After Task 3: Rate ease and effectiveness of Scenario Editor, suggest improvements, evaluate error message clarity
• Final: Complete general feedback section

Data to be collected and evaluation measures

• Quantitative data
  • Likert-scale questions about ease of use for feature set
  • System Usability Scale (SUS) questions
  • Task-specific ease ratings (login, scenario loading, time navigation, task assignment, task re-ordering, scenario editor usage, error message clarity)
• Qualitative data
  • Observed pain points, confusions, and delights
  • Feature requests and suggestions
  • Positive feedback and what's working well
  • Open-ended feedback on overall system experience and suggested improvements

All quantitative and qualitative feedback will be collected through a standardized post-test Google form.

Report contents and presentation

• GitHub wiki page containing all our Usability Testing information
  • Summarizes the background of the study
    • including the goals, methodology, and participant characteristics
  • Present findings for original questions to investigate
  • Give user feedback results and discuss specifics (appropriate to data)
  • Suggest follow-up research
• SOEN 490 Release Presentation videos
  • Summarize usability testing plan and our results
  • Discuss how feedback was incorporated into design and implementation

Usability Testing Results - Round 1

We have maintained an ongoing collaborative feedback loop with our stakeholder throughout development, incorporating their input at multiple touchpoints including design mock-ups, live production testing, and backlog planning. Reaching our second release milestone provided an opportunity to conduct our very first round of structured usability testing. From January 13 to January 17, we engaged friends and family as test users, guiding them through a standardized bilingual script that evaluated core workflows of real BIXI dispatchers. We also had the opportunity to get feedback from 1 domain expert at BIXI (part of the core user base following the end of Release 3).

We collected both quantitative usability metrics and qualitative feedback to meet our initial objectives and answer the research questions described below.

Summary

Participant/Persona Type	Expected Number of Participants	Actual Number of Participants	Experience Level	Notes
Dispatcher	12 (minimum)	22	Domain experts and friends/family	1 domain expert, 21 friends/family debriefed with a script
Scenario Administrator	3 (minimum)	9	Simulated by friends and family	Debriefed with a script

Note: Only 1 domain expert was available to participate during the Release 2 testing window.

Dispatcher Survey Results

Likert Scale Questions (login, load scenario, speed up simulation, assign tasks, re-order tasks):

Dispatcher Task Completion Time Results

TODO

Scenario Admin Survey Results

Likert Scale Questions (usability of scenario editor, helpfulness of error messages):

Open-ended Questions (scenario editor feedback):

Feedback pertaining to the scenario editor:

• Issues due to the use of the JSON format to define the scenarios (clarity, readability, troubleshooting validation issues, too text-heavy) (7 mentions)
  • Alternatives proposed include a different format than JSON, form-based (4), a datetime picker for timestamp fields, map-based (world-builder-esque)
• The scenario editor lacks functionality expected in text editors (tab-friendliness, line numbers, syntax coloring, collapsable fields) (2 mention)
• Difficult to understand what to delete and what not to delete in the json editor (1 mention)
• The scenario doesn't save automatically when pressing Start simulation (1 mention)
• Desire for the ability to drag a JSON file over the editor to import / save it (1 mention)
• The use of red for the selected scenario is confusing since red is usually used to indicate something bad (1 mention)
• The placement of the scenario description field before the scenario content field is unatural (1 mention)

Feedback pertaining to the login page:

• Log in page should allow white-space around the username (1 mention)

Feedback pertaining to the usability test:

• Titles in the task tables should match the json titles (1 mention)
• Instructions could be a little simplified (1 mention)

Bugs discovered:

• Clicking on the current scenario after it has been edited shows a "Unsaved Changes" card (1 mention)

Scenario Admin Task Completion Time Results

TODO

General User Feedback Compilation

TODO

Key Takeaways

TODO

Action Items & Issues Created

TODO