🤖 Jarvis AI Assistant

The Ultimate macOS AI Assistant — Combining the best of ChatGPT, Siri, Claude, Cursor & Cluely

✅ Voice Conversation | ✅ Screen Understanding | ✅ Mac Control | ✅ Memory System | ✅ Planning | ✅ Native macOS UI

---

🚀 Overview

Jarvis is a cutting-edge AI assistant that combines voice conversation, computer vision, screen understanding, and deep Mac automation capabilities. Built with Swift 6/SwiftUI for the frontend and FastAPI/LangGraph for the backend.

New in v2.0

• 🧠 Dual Modes: Reasoning (detailed planning) or Fast (quick responses)
• 📋 Step-by-Step Planning: Visual plan display with real-time status updates
• 🗄️ Memory System: Knowledge graph + vector search for context retention
• 🎙️ Voice Pipeline: Pipecat-inspired real-time voice with interruption handling
• 🎨 Liquid Glass UI: Modern iOS/macOS 26 design language

What makes Jarvis unique:

• 🖥️ Deep macOS Integration — 100+ AppleScripts + 43 AI Tools + Accessibility APIs
• 👁️ Screen Understanding — Capture & analyze any screen, window, or selection
• 🎯 Always-on-Top Focus Mode — Like Cluely, but with full AI capabilities
• 🔒 Privacy-First — Local LLM support, no cloud dependency required

✨ Key Features

🎙️ Voice Conversation Mode

• Natural voice interaction with interruption handling
• Wake word detection — "Hey Jarvis" always listening
• Streaming TTS with premium voices
• Hands-free and push-to-talk modes
• Context-aware conversations with memory

👁️ Vision & Screen Understanding

• Full screen capture — Analyze entire display
• Window capture — Screenshot active window
• Selection capture — Capture specific regions
• Multi-display support — Handle multiple monitors
• Accessibility inspection — Read UI elements, buttons, text fields
• Document processing — PDF, images, text with OCR

🖥️ Mac Automation (100+ Scripts, 43 Tools)

┌─────────────────────────────────────────────────────────────────┐
│                 Mac Automation Categories                        │
├──────────────┬──────────────┬──────────────┬───────────────────┤
│   SYSTEM     │    APPS      │   BROWSER    │   PRODUCTIVITY    │
│ • Battery    │ • Open/Quit  │ • Safari     │ • Calendar        │
│ • Volume     │ • List Apps  │ • Chrome     │ • Reminders       │
│ • Dark Mode  │ • Frontmost  │ • URLs/Tabs  │ • Notes           │
│ • WiFi Info  │ • Hide/Show  │ • Navigation │ • Mail            │
│ • Brightness │ • Switch     │              │ • Messages        │
├──────────────┼──────────────┼──────────────┼───────────────────┤
│   FINDER     │    MEDIA     │  UTILITIES   │   ACCESSIBILITY   │
│ • Navigate   │ • Play/Pause │ • Clipboard  │ • UI Elements     │
│ • Create     │ • Next/Prev  │ • Terminal   │ • Window Info     │
│ • Open Files │ • Track Info │ • Spotlight  │ • Click Buttons   │
│ • Selection  │ • Playlists  │ • Spaces     │ • Type Text       │
│              │              │              │ • Menu Bars       │
├──────────────┴──────────────┴──────────────┴───────────────────┤
│                    SCREEN CAPTURE & VISION                       │
│ • Full Screen  • Window  • Selection  • Multi-Display  • OCR    │
└─────────────────────────────────────────────────────────────────┘

🎯 Interface Modes

Mode	Description	Use Case
Focus Mode	Always-on-top floating panel	Quick access while working
Chat Mode	Full-window conversational UI	Deep conversations
Conversation Mode	Voice-first interface	Hands-free interaction

🔒 Safety & Privacy

• On-device processing with Ollama
• Blocked destructive operations — Cannot delete files
• No password/keychain access
• Local data storage option
• Transparent data usage

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Jarvis AI Architecture                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              Swift 6 / SwiftUI Frontend                  │   │
│  │  ┌──────────┬──────────┬──────────┬────────────────┐    │   │
│  │  │ Chat     │ Focus    │ Voice    │ Settings      │    │   │
│  │  │ Mode     │ Panel    │ Conv.    │ Panel         │    │   │
│  │  └──────────┴──────────┴──────────┴────────────────┘    │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ Services: Audio | Speech | Streaming | API        │  │   │
│  │  └────────────────────────────────────────────────────┘  │   │
│  └─────────────────────────┬───────────────────────────────┘   │
│                            │ HTTP/SSE/WebSocket                 │
│  ┌─────────────────────────▼───────────────────────────────┐   │
│  │                  FastAPI Backend                         │   │
│  │  ┌──────────┬──────────┬──────────┬────────────────┐    │   │
│  │  │ Chat API │Voice API │Files API │  WebSocket    │    │   │
│  │  └──────────┴──────────┴──────────┴────────────────┘    │   │
│  └─────────────────────────┬───────────────────────────────┘   │
│                            │                                    │
│  ┌─────────────────────────▼───────────────────────────────┐   │
│  │               LangGraph Agent Orchestrator               │   │
│  │  ┌──────────────────────────────────────────────────┐   │   │
│  │  │              Tool Router / Planner                │   │   │
│  │  └──────────────────────────────────────────────────┘   │   │
│  │  ┌────────┬────────┬────────┬────────┬────────────┐    │   │
│  │  │ OpenAI │ Ollama │ChromaDB│  Web   │    Mac     │    │   │
│  │  │  API   │ Local  │  RAG   │ Search │ Automation │    │   │
│  │  └────────┴────────┴────────┴────────┴────────────┘    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            Mac Automation Layer (100+ Scripts)            │   │
│  │  ┌──────────┬──────────┬──────────┬────────────────┐    │   │
│  │  │ System   │   Apps   │  Screen  │ Accessibility │    │   │
│  │  │ Control  │  Control │  Capture │   UI Control  │    │   │
│  │  └──────────┴──────────┴──────────┴────────────────┘    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

🚀 Quick Start

Prerequisites

• macOS 13.0+ (Ventura or later)
• Python 3.11+
• Xcode 15.0+
• OpenAI API Key (for GPT models)
• Ollama (optional, for local LLMs)

1. Backend Setup

cd backend

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your OPENAI_API_KEY

# Start server
python main.py

Backend runs on http://localhost:8000

2. Frontend Setup

cd frontend/JarvisAI

# Open in Xcode
open JarvisAI.xcodeproj

# Build and Run (⌘R)

3. Ollama Setup (Optional)

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull models
ollama pull llama3.2
ollama pull llava

# Start Ollama
ollama serve

4. Configuration (Optional)

By default, the app connects to http://localhost:8000. To change this (e.g., for deployment):

• Environment Variable: Set JARVIS_API_URL in your scheme or environment.
• Info.plist: Add a key JarvisAPIURL with your backend URL.

📁 Project Structure

Jarvis/
├── backend/                    # FastAPI Python Backend
│   ├── main.py                # Application entry
│   ├── api/
│   │   └── routes/            # REST endpoints
│   │       ├── chat.py        # Chat streaming
│   │       ├── files.py       # File uploads
│   │       └── conversation.py # Voice handling
│   ├── agents/
│   │   ├── graph.py           # LangGraph workflow
│   │   ├── tools.py           # AI tool definitions
│   │   └── state.py           # Agent state
│   ├── services/
│   │   ├── mac_automation/    # AppleScript engine
│   │   │   ├── executor.py    # Safe script execution
│   │   │   └── scripts.py     # 100+ pre-built scripts
│   │   ├── file_processor/    # Document processing
│   │   └── search_service.py  # Web search
│   └── core/
│       ├── config.py          # Settings
│       ├── openai_client.py   # OpenAI integration
│       └── chroma_client.py   # Vector DB
│
├── frontend/                   # Swift/SwiftUI Frontend
│   └── JarvisAI/
│       ├── JarvisAIApp.swift  # App entry + Focus Panel
│       ├── Views/
│       │   ├── ChatView.swift
│       │   ├── FocusPanelView.swift
│       │   ├── UnifiedPanelView.swift
│       │   └── ConversationModeView.swift
│       ├── ViewModels/
│       │   ├── ChatViewModel.swift
│       │   └── ConversationViewModel.swift
│       └── Services/
│           ├── APIService.swift
│           ├── StreamingService.swift
│           ├── AudioPipeline.swift
│           ├── SpeechRecognitionService.swift
│           └── SpeechSynthesisService.swift
│
├── MAC_CONTROL.md             # Mac automation documentation
└── README.md                  # This file

🎯 Usage Examples

Voice Commands

"Hey Jarvis, what's my battery level?"
"Hey Jarvis, open Safari and go to github.com"
"Hey Jarvis, set volume to 50%"
"Hey Jarvis, what apps are running?"
"Hey Jarvis, toggle dark mode"

Screen Understanding

"Take a screenshot and tell me what you see"
"What's the title of my current window?"
"What buttons are visible on screen?"
"Read the text in the focused field"

Mac Automation

"Create a folder called 'Projects' on my desktop"
"Play the next song"
"What's on my calendar today?"
"Send a notification saying 'Meeting in 5 minutes'"
"Open Terminal and run ls"

⚙️ Configuration

Backend (.env)

OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o
OLLAMA_BASE_URL=http://localhost:11434
CHROMA_DB_PATH=./chroma_db
MAX_FILE_SIZE=10485760

🔧 Available Tools (43 Total)

The AI has access to these tool categories:

Category	Tools	Description
Knowledge	3	search_knowledge_base, web_search, process_uploaded_file
Mac Basic	4	run_mac_script, execute_applescript, execute_shell_command, get_available_mac_scripts
App Control	5	launch_app, quit_app, hide_app, get_running_apps, get_frontmost_app
Browser	9	Navigate, focus, scroll, click, search, get page info
Web Page	5	Get elements, click, fill inputs, execute actions
Input	3	click_at_position, type_text, press_keyboard_shortcut
UI Access	2	get_ui_elements, click_ui_element
Screen	4	capture_screen_for_analysis, wait_seconds, get_screen_text_content, verify_app_is_frontmost
System	6	get_system_state, open_file_or_url, reveal_in_finder, get_current_media_info, send_system_notification, manage_window
Shortcuts	2	run_shortcut, list_shortcuts

See MAC_CONTROL.md for complete documentation.

🛡️ Safety Guardrails

Jarvis blocks all destructive operations:

• ❌ Delete/remove/trash files
• ❌ Empty trash
• ❌ Format/erase disks
• ❌ Shutdown/restart system
• ❌ Access keychain/passwords
• ❌ Modify security settings

🐛 Troubleshooting

Issue	Solution
Microphone not working	System Preferences → Privacy → Microphone
Screen capture fails	Grant screen recording permission
Ollama not connecting	Run ollama serve
Backend won't start	Check port 8000 is free

📄 License

MIT License - see LICENSE for details.

---

Jarvis — Your AI-powered Mac companion

Built with Swift, Python, LangGraph & ❤️