Skip to content

Usability Testing

Jump to bottom
Nicolas Rico edited this page May 20, 2026 · 4 revisions

Usability Testing Protocol

This page describes the planned protocol for Sentinel’s usability testing. It’s important to clarify from the beginning that in Sprint 2 we are only planning the protocol; actual execution with users will take place during Sprint 3, as required by the master document.

Why we conduct usability testing

Automated tests verify that the code does what it is supposed to do, but they tell us nothing about whether an on-call engineer actually understands what is happening when they open the dashboard at 3 a.m.

To answer that question, we need to put real users in front of the product and observe them.

Objective:
Validate that a DevOps or SRE engineer can identify, understand, and resolve an incident without prior training, in less than 5 minutes per flow.

If we don’t achieve this, the product does not solve the problem we promised.

Participants

We will invite 6 participants to ensure feedback triangulation:

  • 3 SRE/DevOps engineers (1–3 years of experience)
    Primary audience

  • 2 senior students with basic Docker experience
    Validate onboarding and usability for juniors

  • 1 session with the Product Owner (SoftServe)
    Validate alignment with client vision (in-person)

Hypotheses

We define four testable hypotheses:

  1. Critical incident detection (<10s)
    Measured by time to first correct click

  2. Root cause understanding (no external docs)
    User explains cause in their own words

  3. Confident decision-making
    No hesitation like “I don’t know what this does”
    Measured via observation + 1–5 rating

  4. Unified flow perception
    Approval → execution → verification feels like one screen
    Validated via final session question

Research Questions

  • Does the layout clearly show:

    • where the user is?
    • which incident is selected?
    • current agent phase?

  • Is the agent reasoning (Markdown):

    • readable?
    • too technical?

  • Is the risk level understandable before approval?

  • Does the timeline:

    • tell a story?
    • or feel like raw logs?

  • Do runbooks:

    • add value?
    • or create noise?

Tasks

Tasks are goal-oriented (not step-by-step).

Task 1: Quick triage

“You just started your on-call shift. Which incident would you handle first and why?”

  • Target: < 1 min
  • Success: selects critical or most recent unattended

Task 2: Understand incident

“Explain what happened in app-demo and what the system proposes.”

  • Target: < 2 min
  • Success:
    • incident type
    • root cause
    • at least one action

Task 3: Approve action

“The system proposes restarting the container. Approve or reject.”

  • Target: < 1 min
  • Success:
    • reads risk
    • reviews command
    • leaves comment
    • makes informed decision

Task 4: Create incident

“Create an incident for a failure in the customers database.”

  • Target: < 2 min
  • Success:
    • title
    • target: postgres/customers
    • severity: high
    • saved

Task 5: Find similar incidents

“Check if this incident happened before.”

  • Target: < 1 min
  • Success:
    • finds similar incidents section
    • opens one

Evaluation Metrics

Quantitative

  • Task success rate ≥ 80%
  • Time within targets
  • Max 1 critical error per session

Standard Metrics

  • SUS (System Usability Scale)
    Target: ≥ 70

  • NPS (0–10)
    Question: “Would you recommend this tool to a colleague?”
    Target: ≥ 7

Qualitative

  • Thematic analysis (patterns, not anecdotes)

Session Logistics

  • Platform: Microsoft Teams (screen sharing)
  • Duration: 45 minutes
Phase Time
Introduction 5 min
Tasks 25 min
Questionnaire 10 min
Open questions 5 min
  • Sessions are recorded (audio + screen) with consent

Roles

  • Moderator: Nicol Garcia Tabares
  • Observer: Jacobo Montes

Roles are separated to maintain quality.

Environment

  • Staging instance
  • 10 seeded incidents
  • Same dataset for all participants

Execution: Week of May 13, 2026 (Sprint 3)

Moderator Script

  1. Introduction (2 min)

    • Explain context
    • Clarify: we test the product, not the user

  2. Consent

  3. Background questions (3 min)

    • Experience with:
      • PagerDuty
      • Datadog
      • Grafana

  4. Tasks (25 min)

    • Think-aloud method

  5. Questionnaire (5 min)

    • SUS + NPS

  6. Open feedback (5 min)

    • What would you change?
    • What’s missing?
    • Would you use it tomorrow?

Analysis Plan

After sessions:

  • Consolidate metrics in spreadsheet
  • Cross with user quotes

Output

  • 3–5 actionable insights
  • Prioritized:
    • High
    • Medium
    • Low

Presented in Sprint 3 retrospective.

Pre-Execution Checklist

  • Seeded data in Supabase (10 incidents)
  • Test accounts created
  • Observer notes template
  • SUS form (Google Forms)
  • Sessions scheduled
  • Signed consent forms

Clone this wiki locally