Walkable

AI-powered software teaching overlay for macOS. Walkable watches your screen and teaches you any application through live highlights, step-by-step instructions, and adaptive learning paths.

How it works

1. Pick an app — choose from the catalogue (Ableton, Figma, Blender, VS Code, etc.) or describe what you want to learn
2. AI generates a plan — a structured course with modules and steps, tailored to your experience level
3. Learn with live overlays — the AI captures your screen, highlights UI elements, and walks you through each step
4. Adapt on the fly — ask questions mid-course and the AI branches your learning path dynamically

Architecture

• SwiftUI app with @Observable state management
• Screen capture via ScreenCaptureKit for live vision
• LLM tool calling — the AI controls the entire UX through tools (highlight_region, show_step, ask_user, update_plan, etc.)
• Floating overlay panel + full-screen transparent highlight window
• Course dashboard with module progress, chat, and changelog tree

Requirements

• macOS 15+
• Screen Recording permission
• Accessibility permission
• API key for OpenAI or Anthropic (configured in ~/.walkable/config.json)

Setup

# Create config
mkdir -p ~/.walkable
cat > ~/.walkable/config.json << 'EOF'
{
  "provider": "anthropic",
  "apiKey": "sk-ant-...",
  "model": "claude-sonnet-4-20250514"
}
EOF

# Build and run
cd Walkable
swift build
swift run Walkable

Project structure

Walkable/
├── Sources/Walkable/
│   ├── WalkableApp.swift          # App entry point, window management
│   ├── Engine/
│   │   ├── LearningEngine.swift   # LLM tool loop + vision pipeline
│   │   └── SystemPrompt.swift     # Teaching behavior instructions
│   ├── Models/
│   │   ├── OverlayState.swift     # Observable state, data types
│   │   └── Configuration.swift    # Provider config
│   ├── Views/
│   │   ├── LandingView.swift      # App catalogue + discovery
│   │   ├── CourseOverviewView.swift # Module dashboard + chat + changelog
│   │   ├── OverlayContentView.swift # Floating teaching panel
│   │   └── HighlightOverlayView.swift # Screen highlight rectangles
│   ├── ViewModels/
│   │   └── WalkableViewModel.swift
│   ├── Tools/
│   │   └── WalkableTools.swift    # LLM tool definitions
│   ├── Services/
│   │   ├── AppDetector.swift      # Running app discovery
│   │   └── ScreenCapture.swift    # Screenshot pipeline
│   ├── Overlay/
│   │   ├── OverlayPanel.swift     # NSPanel subclass
│   │   └── OverlayWindowController.swift
│   └── Utilities/
│       ├── Log.swift
│       └── Permissions.swift
└── Package.swift