Skip to content

Add desktop E2E flows, feature vector, and flow-walker reports#5848

Merged
beastoin merged 24 commits intomainfrom
sora/desktop-e2e-flows
Mar 20, 2026
Merged

Add desktop E2E flows, feature vector, and flow-walker reports#5848
beastoin merged 24 commits intomainfrom
sora/desktop-e2e-flows

Conversation

@beastoin
Copy link
Copy Markdown
Collaborator

@beastoin beastoin commented Mar 20, 2026

Summary

  • Add desktop/e2e/feature-vector.md — 31 desktop features scored by priority × walker_score
  • 7 flow YAMLs in v2 format: navigation, dashboard, chat, memories, tasks, settings, language
  • All flows use beta bundle ID (com.omi.computer-macos)
  • Flow-walker snapshot caches for replay optimization
  • Updated SKILL.md with v2 format, report links, and corrected guard conditions

Flow-Walker Reports (fresh runs, Phase 11)

Flow Steps Result Report
navigation 6/6 PASS report
dashboard 3/6 PASS (3 skipped) report
chat 5/5 PASS report
memories 5/6 PASS (1 skipped) report
tasks 4/5 PASS (1 skipped) report
settings 9/9 PASS report

Total: 32 pass, 5 skipped, 0 failed across 6 flows

Skipped Steps (AX accessibility gaps)

  • Dashboard: goal progress slider, task checkboxes, goal delete — custom SwiftUI views without AX exposure
  • Memories: tag filter chips — not exposed via AX API
  • Tasks: task completion checkbox — not exposed via AX API

Issues Found

  • Chat backend returned "Something went wrong" on Mac Mini (flagged to ren)

Test plan

  • All 6 flows run fresh with flow-walker Phase 11 (record init → stream → finish → verify → report → push)
  • All reports published with screenshots at each step
  • SKILL.md updated with v2 format and report links

🤖 Generated with Claude Code

beastoin and others added 2 commits March 20, 2026 02:39
31 features scored by priority × walker_score, 24 covered, 7 gaps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard → Chat → Memories → Tasks → Apps → Settings.
Uses AX identifiers (sidebar_dashboard, sidebar_chat, etc).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 20, 2026

Greptile Summary

This PR adds a desktop E2E feature vector document and migrates the navigation.yaml flow to the v2 flow-walker format. The feature vector provides a useful prioritized map of 31 desktop features with walker automability scores, and the YAML conversion properly adopts quoted do: strings and switches the target bundle from the dev app (com.omi.desktop-dev) to the beta app (com.omi.computer-macos).

Key issues found:

  • Missing Rewind step in navigation.yaml: The flow description claims "all 7 sidebar sections" but only 6 sections are tested (Dashboard, Chat, Memories, Tasks, Apps, Settings). The Rewind section (sidebar_rewind) — a documented AX identifier — has no corresponding step.
  • Feature vector row Add printing BLE device address to serial monitor #3 incorrectly marked ✅: "Sidebar navigation — all 7 sections" is listed as fully covered by navigation.yaml, which is inaccurate given the missing Rewind step. It should be ⚠️ partial.
  • Remaining Gaps table is incomplete: The Coverage Summary correctly identifies 7 total gaps, but the Remaining Gaps table only enumerates 5. "Refer a Friend" (feature fix: Fix device search for Android #18, ❌ needs flow) and "Quick Note partial" (feature Restructuring #10, ⚠️ partial) are counted in the summary totals but absent from the ranked gap table.

Confidence Score: 3/5

  • Safe to merge with minor documentation accuracy fixes needed — no production code is affected.
  • Both files are documentation/testing artifacts with no impact on runtime behavior. The v2 YAML conversion is functionally correct and the flow-walker report shows 6/6 passing. However, there are meaningful factual inconsistencies: the navigation flow description and feature vector coverage status both claim "all 7 sections" when only 6 are tested, and the Remaining Gaps table is missing 2 of the 7 identified gaps. These inaccuracies could mislead future contributors about actual E2E coverage state.
  • Both desktop/e2e/feature-vector.md and desktop/e2e/flows/navigation.yaml have the same root discrepancy around the claimed vs. actual sidebar section coverage count.

Important Files Changed

Filename Overview
desktop/e2e/flows/navigation.yaml Converted from v1 to v2 flow-walker format with cleaner do: string steps; the bundle_id changed from com.omi.desktop-dev (dev) to com.omi.computer-macos (beta). However, the description claims "all 7 sidebar sections" while only 6 steps are present — the Rewind section (sidebar_rewind) is not tested despite the AX identifier being documented.
desktop/e2e/feature-vector.md New feature vector document covering 31 desktop features with priority and walker scores. The Coverage Summary correctly tallies 7 gaps, but the Remaining Gaps table only lists 5 entries — "Refer a Friend" and "Quick Note (partial)" are acknowledged in the summary table but omitted from the gap ranking table. Feature #3 is incorrectly marked ✅ complete for "all 7 sections" when navigation.yaml only covers 6.

Sequence Diagram

sequenceDiagram
    participant FW as flow-walker (agent-swift)
    participant App as Omi Desktop (com.omi.computer-macos)

    FW->>App: S1 — Verify Dashboard loads
    App-->>FW: text_visible: Dashboard, interactive_count ≥ 10

    FW->>App: S2 — Click sidebar_chat
    App-->>FW: text_visible: Chat

    FW->>App: S3 — Click sidebar_memories
    App-->>FW: text_visible: Memories

    FW->>App: S4 — Click sidebar_tasks
    App-->>FW: text_visible: Tasks

    FW->>App: S5 — Click sidebar_apps
    App-->>FW: text_visible: Apps

    FW->>App: S6 — Click sidebar_settings
    App-->>FW: text_visible: Settings

    note over FW,App: sidebar_rewind not tested (gap)
Loading

Last reviewed commit: "Add desktop navigati..."

version: 2
name: navigation
description: Desktop app navigation — sidebar icons, section switching, text input, scroll, tray menu
description: Desktop sidebar navigation — verify all 7 sidebar sections load correctly via click navigation
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Description claims "all 7 sidebar sections" but only 6 are tested

The description says "verify all 7 sidebar sections load correctly" but the flow only has 6 steps covering: Dashboard, Chat, Memories, Tasks, Apps, and Settings. The Rewind section (sidebar_rewind) is completely absent from the flow.

Looking at the SidebarView.swift enum, rewind is a valid nav item with identifier clock.arrow.circlepath. The Desktop-Specific Notes in feature-vector.md also confirm sidebar_rewind as a valid AX identifier.

Either the description should be updated to "6 sidebar sections" or a step for Rewind should be added:

Suggested change
description: Desktop sidebar navigation — verify all 7 sidebar sections load correctly via click navigation
description: Desktop sidebar navigation — verify 6 sidebar sections load correctly via click navigation

|---|---------|-------|----------|--------|-----------------|
| 1 | Dashboard — goals widget, tasks, conversations | intelligence (3) | 9 | 3 | ✅ flow: dashboard.yaml (6 steps) |
| 2 | Chat — send message, AI response, actions | intelligence (3) | 9 | 2 | ✅ flow: chat.yaml (5 steps) |
| 3 | Sidebar navigation — all 7 sections | retrieval-action (3) | 9 | 3 | ✅ flow: navigation.yaml |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Feature marked ✅ "all 7 sections" but flow only covers 6

Row #3 marks "Sidebar navigation — all 7 sections" as ✅ covered by navigation.yaml, but the navigation.yaml flow only steps through 6 sections (Dashboard, Chat, Memories, Tasks, Apps, Settings). The Rewind section is missing from the flow.

This creates a false impression of complete coverage. Consider either:

  • Adding a Rewind step to navigation.yaml (the sidebar_rewind AX identifier is already noted in the Desktop-Specific Notes section), or
  • Updating the coverage status to ⚠️ partial to reflect the missing Rewind step
Suggested change
| 3 | Sidebar navigation — all 7 sections | retrieval-action (3) | 9 | 3 | flow: navigation.yaml |
| 3 | Sidebar navigation — all 7 sections | retrieval-action (3) | 9 | 3 | ⚠️ partial: flow: navigation.yaml (6/7 sections, missing Rewind) |

Comment on lines +79 to +88

| Rank | Feature | Priority | Blocker | Notes |
|------|---------|----------|---------|-------|
| 1 | Screen capture (Rewind) | 15 | Needs screen recording permission | Rewind page exists, AX identifiers unknown — needs exploration |
| 2 | Audio recording (desktop mic) | 15 | Needs microphone permission grant | Start Recording button visible on dashboard but mic requires OS dialog |
| 3 | Apps/Integrations | 6 | No flow written | Apps page accessible via sidebar_apps — should be straightforward |
| 4 | Onboarding | 5 | Needs fresh/reset state | Reset Onboarding available in tray menu but causes state corruption (known issue) |
| 5 | Auth | 5 | External OAuth | Google/Apple Sign-In opens browser — not automatable |

---
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Remaining Gaps table is incomplete — 2 of 7 gaps are missing

The Coverage Summary at the bottom of this file correctly identifies 7 total gaps, but the Remaining Gaps table only lists 5 items. The two missing gaps are:

  • Feature Restructuring #10 — Quick Note (dashboard): Listed as ⚠️ partial in the CORE DAILY table and counted in the "3 gaps" for Core Daily (the note says "quick note partial"), but not mentioned in the Remaining Gaps table.
  • Feature fix: Fix device search for Android #18 — Refer a Friend: Listed as ❌ needs flow in the CORE WEEKLY table and counted in the "2 gaps" for Core Weekly, but is missing from the Remaining Gaps table.

Both should be added to keep the Remaining Gaps table consistent with the Coverage Summary. For example:

| 4 | Quick Note (dashboard) | 9 | Partial coverage only | quick_note step exists in dashboard.yaml but not independently verified |
| 5 | Refer a Friend | 6 | No flow written | sidebar_refer_a_friend AX identifier is available |

beastoin and others added 22 commits March 20, 2026 03:48
navigation 6/6, dashboard 3/6, chat 4/5, memories 5/6, tasks 5/5, settings 9/9.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Accurate pass/fail/skipped results with updated verifier.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 flows re-run from scratch: 32 pass, 5 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@beastoin beastoin changed the title Add desktop E2E feature vector and fix navigation flow Add desktop E2E flows, feature vector, and flow-walker reports Mar 20, 2026
Copy link
Copy Markdown
Collaborator Author

@beastoin beastoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@beastoin beastoin merged commit 9d89afa into main Mar 20, 2026
2 checks passed
@beastoin beastoin deleted the sora/desktop-e2e-flows branch March 20, 2026 08:33
Glucksberg pushed a commit to Glucksberg/omi-local that referenced this pull request Apr 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant