Jarvis is a local-first desktop AI assistant for macOS.
The simple version:
- you talk to it
- it understands your request
- it looks up the right memory, tools, and context
- it answers or acts on your Mac
The ambitious version:
Jarvis is trying to become your own private, open-source-first desktop intelligence layer. Not just a chat box. Not just a prompt wrapper. A real system that can think, listen, see, remember, inspect code, control tools, and keep improving over time.
If you are new to this repo, read these sections in order:
What Is Jarvis?What's New (June 2026)How Jarvis WorksCurrent StateRoadmapRepo Map
Imagine a helpful robot friend living on your computer.
That friend should be able to:
- hear you
- talk back
- look at your screen
- remember important things
- help with coding
- use tools on your Mac
- stay private by running locally whenever possible
That is what Jarvis is meant to be.
For engineers, the more exact definition is:
Jarvis is a local-first assistant runtime with:
- a desktop app
- a local API
- model routing
- memory and retrieval
- skills, connectors, and plugins
- a managed task runtime
- multimodal perception
- local-model tuning and eval loops
Most AI products today depend on cloud models as the foundation.
Jarvis is trying to invert that:
- local by default
- cloud only as fallback or explicit escalation
- grounded in your files, tools, and environment
- honest about what it knows and what it does not know
Jarvis is a personal, single-user, local-first assistant runtime. It is built to run on one Mac, for one person, with that person's memory, files, and tools.
What it is for:
- a daily driver assistant that knows your context and runs mostly on local models
- a managed agent team that can research, write code, and run commands on your machine — inside a sandbox, with every write surfaced for your review
- a test bed for local-model routing, memory architecture, and safe agent-runtime design
What it is not:
- not a hosted service or multi-tenant product
- not an autonomous system — agents never merge code, never write to your knowledge vault, and never persist "lessons" without explicit human approval
- not cloud-dependent —
open-sourcemode keeps core inference on local models only
Two waves shipped this month. The second wave turned the agent team into a real runtime with tools, isolation, and verification:
-
Agents With Real Tools — Agents now run a multi-turn tool loop instead of producing one-shot text. They get an allowlist-validated
bashtool (list-args,shell=False, pipelines supported with each stage validated separately; redirection,;, and$()are rejected) plus a tool budget that forces a final no-tools answer when exhausted. -
Seatbelt Sandbox on Every Command — Every agent command runs under macOS
sandbox-exec: network denied, file writes confined to the task worktree and tmp, inherited by child processes. Live-proven: agent-spawned test code getsPermissionErrorwriting to$HOMEand denied socket connects, while worktree writes succeed. -
Isolated Code Worktrees —
kind="code"tasks get their own git worktree on acodex/<task>branch. Thewrite_filetool can only write inside it. The resulting diff is staged and surfaced for human review — the runtime never merges a branch. E2E-proven: an agent wrote a module plus tests, watched pytest fail, fixed its own bug, and surfaced the passing diff. -
Auto-Verification With Fabrication Defense — Task output is scored by a verifier that receives the runtime-captured tool transcript as ground truth. If an agent claims it executed something but the runtime counted zero tool calls, the output is flagged before verification. Failed tasks retry (max 2).
-
Routines and Approval-Gated Lessons — Cron-style routines run agents on a schedule (
/routinesCRUD, RBAC-protected). Agents can propose lessons learned, but nothing persists until a human approves it in the dashboard. -
Context Minimalism — Multi-task waves chain through a compact result index; downstream agents pull full upstream results only when they ask for them.
-
Local OSINT Intelligence Engine — Four local tools (Maigret username scan, DNSTwist typosquatting, subfinder passive subdomain enumeration, system WHOIS) wired into Jarvis as first-class tools. All results cached in SQLite with per-tool TTL. Heavy scans rate-limited (5/60s per caller). A single
/osint/worldviewendpoint aggregates whois + subdomains in parallel (typos opt-in viainclude_typos). Full TypeScript service layer in WorldView (enumSubdomains,whoisLookup,worldviewScan) and Vercel edge proxy. 42 unit tests. WorldViewJarvisOsintPanelrenders scan results inline.
Earlier in June:
- 14-Agent Specialized Team — A named roster of purpose-built agents runs inside the task runtime. Each agent has a defined role:
backend_engineer,researcher,security_reviewer,automation_engineer,devops_release,frontend_designer,ux_researcher,qa_tester,ai_evaluator,ai_safety_agent,pipeline_monitor,output_quality_checker,career_agent,memory_librarian. Dispatch any task to any agent viaPOST /tasks. - Three.js 3D Office World — The
/dashboardnow renders a live Roblox-style 3D office in WebGL. Each agent appears as an R6 avatar character with named animations: idle breathing, walk cycle, typing at desk, gym lifting, done jump, failed droop, speaking gesture, waiting head-sway. The office is zoned by function (engineering, design, security, gym, meeting room) with isometric camera. Click an agent to inspect its state. - Single Unified Dashboard — Two dashboards (
/mobile HUD and/dashboardops panel) are now one. The root/redirects to/dashboard, eliminating a 2038-line duplicate UI surface. - Fast-Path Contamination Fix — Agent tasks were being silently intercepted by keyword fast-paths in
router.py(task list, calendar routes) before ever reaching the LLM, producing garbage output in ~26ms. Fixed by dispatching agent tasks directly throughsmart_streaminmodel_router.pywith an agent-specific system context.
Five major systems landed this month:
- Always-On Brain Core — Loads your identity, projects, preferences, and roadmap from vault notes as a permanent system prefix. Every model call knows who you are and what Jarvis is for.
- Parallel Agent Layer — ThreadPoolExecutor fan-out to specialized sub-agents for calendar, tasks, vault, code, and research. Commands: "brief me", "what's my status", "what needs my attention".
- mem0 Cross-Session Memory — Ollama-embedded local Qdrant vector store. Persists conversation facts across sessions. Degrades silently when Ollama unavailable.
- Qwen3 Model Fleet — Qwen3 8b (general), Qwen3 30b-a3b (near-GPT4), Devstral (coder), phi4-mini (fast). Join Deepseek R1 for reasoning work.
- 4-Tier Memory System — Vault (Obsidian brain) + graph context + semantic TF-IDF/embed + mem0 episodic. Parallel context sources in smart_stream.
flowchart LR
User["You"] --> UI["Desktop UI / CLI"]
UI --> API["Local API\n/dashboard → 3D Office World"]
API --> Router["Router + Orchestrator"]
Router --> Brain["Brain Core\nIdentity + Preferences"]
Router --> Memory["Memory + Vault\n+ Graph + Semantic + mem0"]
Router --> Agents["Parallel Agents\nCalendar / Tasks / Code"]
Router --> Skills["Skills"]
Router --> Tools["Tools + Connectors"]
Router --> OSINT["OSINT Engine\nWorldview · Username · Subdomains · WHOIS"]
Router --> Models["Local Models\nsmart_stream direct"]
Router --> Tasks["Managed Task Runtime\n14 Specialist Agents"]
Tasks --> AgentTeam["backend · researcher · security\nautomation · devops · frontend\nux · qa · ai_eval · ai_safety\npipeline · quality · career · memory"]
AgentTeam --> Sandbox["Sandboxed Tool Loop\nbash + write_file · Seatbelt\ngit worktrees · verification"]
Tools --> Mac["Browser / Terminal / macOS / Devices"]
Every request follows roughly this path:
- You ask Jarvis something.
- Jarvis loads your identity from Brain Core.
- Jarvis decides what kind of request it is.
- Jarvis loads the right skills and context (4-tier memory system).
- Jarvis chooses the right model and tools.
- Jarvis can dispatch parallel agents if useful.
- Jarvis answers, acts, or starts a managed task.
- Managed tasks run agents in a sandboxed tool loop, and their output is verified against what they actually executed.
- Jarvis stores useful memory for future sessions.
This is the visible Jarvis window and compact shell.
Main files:
What it does:
- shows the UI
- starts the local API/runtime
- lets you talk to Jarvis like a desktop product instead of a terminal script
This is the shared runtime surface that both the desktop app and CLI can talk to.
Main file:
What it does:
- exposes chat, memory, runtime, task, skills, connectors, and plugin endpoints
- makes Jarvis act like a real local service instead of a one-off script
This is the decision-making layer before any model answers.
Main files:
What it does:
- figures out what kind of request you made
- decides whether Jarvis should answer directly, use a skill, or start a task
- chooses the model path
- attaches relevant context first
This is how Jarvis avoids feeling generic.
Main files:
What it does:
- stores facts and preferences
- searches local markdown knowledge
- grounds repo questions in generated graph artifacts
- persists episodic memory across sessions via mem0
- gives Jarvis 4-tier parallel context across sessions
This is your identity snapshot baked into every model call.
Main file:
What it does:
- loads Identity, Projects, Preferences, and Roadmap notes from vault at startup
- prepends a ~1400-char system context to every model call
- plays the same role for Jarvis that CLAUDE.md plays for Claude Code
- ensures every response knows who Aman is and what matters to you
This is how Jarvis can multitask intelligently.
Main file:
What it does:
- dispatches specialized sub-agents to calendar, tasks, vault, code, and research in parallel
- responds to meta-commands: "brief me", "morning briefing", "what's my status", "what needs my attention", "run agents on X"
- uses ThreadPoolExecutor to avoid blocking
- escalates complex queries to the main model when needed
This is episodic memory that persists across sessions.
Main file:
What it does:
- Ollama-embedded, local Qdrant vector store at
~/.mem0/jarvis/ - fourth parallel context source in smart_stream
- stores conversation facts and observations
- retrieves relevant past context automatically
- degrades silently if Ollama becomes unavailable
This is Jarvis's extensibility layer.
Main files:
What each one means:
Skills: instruction packs for specific types of workConnectors: integrations into real capabilities like browser, terminal, vault, and Google WorkspacePlugins: bundles of skills, connectors, and agents that feel like complete features
This is the part that lets Jarvis act more like an operator than a chatbot.
Main files:
- task_runtime.py
- agents/ — task-based agent modules
- jarvis_daemon.py
- worktree_manager.py
What it does:
- dispatches tasks to 14 specialized named agents, each backed by a direct
smart_streamLLM call with role-specific system context - runs each agent in a multi-turn tool loop: allowlist-validated
bash(list-args,shell=False, pipeline stages validated separately) pluswrite_file, under a tool budget - wraps every agent command in a macOS Seatbelt sandbox — network denied, writes confined to the task worktree and tmp, inherited by child processes
- gives
kind="code"tasks an isolated git worktree; the diff is surfaced for human review and the branch is never auto-merged - verifies output against the runtime-captured tool transcript, detects fabricated execution claims, and retries failed tasks (max 2)
- persists projects, events, and routines in SQLite (
~/.jarvis/projects.db); a scheduler thread runs cron-style routines - creates and tracks tasks with status, streaming output, and cancellation
- serializes execution through a single lock (tasks run sequentially, preventing resource contention)
The 14 agents in the current roster:
| Agent | Role |
|---|---|
backend_engineer |
API, databases, services, infrastructure code |
researcher |
Deep research, synthesis, literature review |
security_reviewer |
Threat modeling, vulnerability analysis, hardening |
automation_engineer |
Scripts, CI/CD, workflow automation |
devops_release |
Deployments, releases, infra changes (always sets needs_review=True) |
frontend_designer |
UI components, layouts, design systems |
ux_researcher |
User experience, usability, UX research |
qa_tester |
Test plans, test cases, regression coverage |
ai_evaluator |
Model evaluation, evals design, benchmark review |
ai_safety_agent |
Safety analysis, alignment review, risk assessment |
pipeline_monitor |
Pipeline health, bottlenecks, observability |
output_quality_checker |
Output review, quality gates, consistency checks |
career_agent |
Career planning, skill gaps, professional growth |
memory_librarian |
Memory hygiene, knowledge organization, vault curation |
This is the multimodal layer.
Main files:
What it does:
- speech input
- speech output
- meeting assist
- screen and camera understanding
- browser control
- nearby device awareness
Jarvis as of June 2026:
Agent runtime — 14 specialist agents with a sandboxed multi-turn tool loop, isolated git worktrees for code tasks, transcript-grounded verification, and scheduled routines.
OSINT — Local intelligence engine: username footprint (Maigret), typosquatting (DNSTwist), passive subdomain enumeration (subfinder), WHOIS. Aggregated worldview scan completes in ~10–30s. Rate-limited, SQLite-cached, integrated into WorldView frontend via Vercel edge proxy.
Voice — Kokoro TTS with macOS say fallback. Faster-whisper STT (upgrade path: large-v3-turbo).
Memory — 4-tier parallel: vault (Obsidian brain) + graph context + semantic TF-IDF/embed + mem0 episodic.
Agents — Parallel fan-out to calendar, tasks, vault, code, research with escalation detection.
Brain — Always-on identity snapshot from vault brain notes in every model call.
Models — Qwen3 fleet (4b/8b/30b-a3b), Devstral coder, Deepseek R1 reasoning, phi4-mini lightweight.
Capability Overview — Answer questions, explain concepts, plan work, read code, debug problems, remember facts/preferences/projects, ground answers in repo structure and vault, retrieve episodic memory across sessions, control browser/terminal, use skills and tools, create and manage tasks.
| Area | What Jarvis can do now |
|---|---|
| Chat | Answer questions, explain concepts, plan work, and reason about technical topics |
| Coding | Read the repo, explain code, debug problems, review code, and run managed coding tasks |
| Voice | Use local STT (faster-whisper) and local TTS (Kokoro → say fallback) in main path |
| Memory | 4-tier: vault, graph context, semantic embed, and mem0 episodic persistence |
| Brain | Always-on identity snapshot from vault notes in every model call |
| Agents | Parallel fan-out to calendar, tasks, vault, code, research with escalation |
| Repo grounding | Use Graphify and vault-based context instead of pure guesswork |
| Browser + system | Read pages, click controls, open apps, change settings, and take screenshots |
| Managed runtime | Dispatch tasks to 14 named specialist agents, stream output, inspect status, cancel tasks |
| Agent tools | Multi-turn tool loop with allowlisted bash (pipelines supported) and worktree-confined file writes |
| Agent safety | Seatbelt sandbox on every command (no network, confined writes), human-review diffs, never auto-merge |
| Verification | Score agent output against the runtime tool transcript, detect fabricated execution, auto-retry |
| Routines | Schedule agents on cron-style routines; approval-gated lessons for self-improvement |
| 3D Office World | Live WebGL dashboard with R6 avatar agents, zone floors, animated states, click-to-inspect |
| OSINT | Local intelligence engine: username footprint (Maigret), typosquatting (DNSTwist), subdomains (subfinder), WHOIS — rate-limited, SQLite-cached, aggregated via /osint/worldview |
| Extensions | Expose discoverable skills, connectors, and plugins through API and CLI |
Jarvis is configured around a local-first stack:
- default local chat:
qwen3:8b(strong general-purpose) - near-GPT4 quality:
qwen3:30b-a3b - deeper reasoning:
deepseek-r1:14b - local coding:
devstral - fast lightweight:
phi4-mini - local STT:
faster-whisper(upgrade path:large-v3-turbo) - local TTS: Kokoro with macOS
sayfallback - local embeddings:
nomic-embed-text - local vision:
llava:7b
You can inspect the live runtime stack with:
curl http://127.0.0.1:8765/local/capabilitiesThis project is strong enough to be useful now, but it is not done.
Important truth:
- Jarvis is local-first today
- Jarvis defaults to
open-sourcemode, which keeps core inference on local models only - paid fallbacks can still exist outside open-source mode for users who explicitly want them
- the remaining work is mostly about making every subsystem consistently local-quality, not re-centering the stack around cloud APIs
That honesty matters because the goal here is not marketing. The goal is to build the real thing.
Jarvis is moving through a few clear stages.
Goal:
Make Jarvis a real long-lived assistant runtime, not just a window that happens to call a model.
Main work:
- move more ownership into jarvis_daemon.py
- keep runtime_state.py as the shared source of truth
- make UI and CLI act like clients of the runtime
Goal:
Make the default path local for chat, coding, speech, retrieval, and vision.
Main work:
- strengthen model_router.py
- keep local models as the main path
- reduce or remove cloud dependency from meeting, vision, and retrieval paths
Goal:
Make Jarvis answer from memory, repo structure, and real observations instead of fluent guessing.
Main work:
- improve semantic_memory.py
- unify memory, vault, graph, and mem0 grounding
- add stronger retrieval and reranking
Goal:
Let Jarvis act safely and reliably on the machine.
Shipped:
- sandboxed agent tool loop (Seatbelt: no network, confined writes)
- isolated git worktrees with human-review diffs for code tasks
- transcript-grounded verification for multi-step actions
Remaining:
- cleaner connector boundaries
- clearer safe vs privileged tool categories outside the task runtime
Goal:
Make Jarvis feel like a real desktop assistant, not just a text engine.
Main work:
- better local meeting assist
- stronger local screen and camera understanding
- better device and environment awareness
Goal:
Let Jarvis get better through evals and controlled local-model improvement.
Main work:
- stronger eval suites
- safer self-improvement loops
- better local-model training and promotion workflows
If the repo feels big, this is the shortest useful map:
| Path | Purpose |
|---|---|
| main.py | App startup |
| ui.py | Main desktop UI |
| api.py | Local API surface |
| router.py | Main request routing |
| model_router.py | Model choice and mode policy |
| jarvis_core_brain.py | Always-on identity snapshot |
| jarvis_agents.py | Parallel agent dispatcher |
| mem0_layer.py | Cross-session episodic memory |
| memory.py | Stored facts and profile memory |
| semantic_memory.py | Retrieval over memory data |
| vault.py | Local markdown knowledge vault |
| graph_context.py | Repo graph grounding |
| voice.py | Speech in and speech out |
| meeting_listener.py | Meeting assist |
| camera.py | Screen and camera understanding |
| browser.py | Browser control |
| hardware.py | Mac and device awareness |
| task_runtime.py | Managed task system — 14-agent team, streaming, cancellation |
| agents/ | Task-based agent modules (backend_engineer, researcher, etc.) |
| _bg_agents.py | Legacy background agents (EmailAgent, start/stop) |
| skills/ | Skill packs |
| connectors/ | Connector definitions |
| plugins/ | Plugin definitions |
| docs/jarvis_architecture/ | Architecture docs and roadmap |
Jarvis now exposes a Claude-style discovery layer:
./venv/bin/python jarvis_cli.py --skills
./venv/bin/python jarvis_cli.py --connectors
./venv/bin/python jarvis_cli.py --plugins
curl http://127.0.0.1:8765/extensionsCore runtime:
GET /statusGET /runtime/statePOST /chatGET /modePOST /mode
Extensions:
GET /extensionsGET /skillsGET /skills/{skill_id}GET /connectorsGET /connectors/{connector_id}GET /pluginsGET /plugins/{plugin_id}
OSINT local endpoints:
GET /osint/status— tool availability (maigret, dnstwist, subfinder, whois)POST /osint/username— username footprint scan via MaigretPOST /osint/domain-typos— typosquatting detection via DNSTwistPOST /osint/subdomains— passive subdomain enumeration via subfinderPOST /osint/whois— WHOIS registration lookupPOST /osint/worldview— aggregated domain/username scan (whois + subdomains in parallel; typos opt-in)GET /osint/cache— inspect cached scan resultsDELETE /osint/cache— clear cache by tool name
Managed runtime:
GET /agentsGET /tasksPOST /tasksGET /tasks/{task_id}GET /tasks/{task_id}/eventsGET /tasks/{task_id}/stream
Routines and lessons (RBAC-protected):
GET /routinesPOST /routinesDELETE /routines/{routine_id}GET /agents/{agent_id}/lessons/pendingPOST /agents/{agent_id}/lessons/approvePOST /agents/{agent_id}/lessons/dismiss
Memory and vault:
GET /memoryGET /memory/statusPOST /memory/addPOST /memory/forgetGET /vaultPOST /vault/build
mem0 endpoints:
GET /mem0/statusPOST /mem0/addGET /mem0/search
If ~/.local/bin is on your PATH, use Jarvis like a console-native assistant:
jarvis
jarvis --doctor
jarvis --parity
jarvis --capability-evals
jarvis --security-roe ai
jarvis --code-ultra "inspect the auth middleware and propose the smallest safe patch"The jarvis wrapper resolves the project venv before starting the daemon, so local dependencies such as faster-whisper stay available even if your shell's default python3 points at Anaconda or another base interpreter.
./venv/bin/python jarvis_cli.py --status
./venv/bin/python jarvis_cli.py --skills
./venv/bin/python jarvis_cli.py --connectors
./venv/bin/python jarvis_cli.py --plugins
./venv/bin/python jarvis_cli.py "explain optimistic locking like I'm 10"
./venv/bin/python jarvis_cli.py --task "summarize the current repo architecture"
./venv/bin/python jarvis_cli.py --task-code "refactor the auth middleware"
./venv/bin/python jarvis_cli.py "brief me"
./venv/bin/python jarvis_cli.py "what needs my attention"git clone https://github.com/amanimran786/jarvis-ai.git
cd jarvis-ai
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtbrew install ollama
brew services start ollamaollama pull qwen3:8b # strong general-purpose (recommended)
ollama pull qwen3:30b-a3b # near-GPT4 quality on Mac
ollama pull gemma4:31b # Gemma 4 workstation eval candidate
ollama pull gemma4:26b # Gemma 4 MoE eval candidate
ollama pull qwen3.6:35b # Qwen3.6 local eval candidate
ollama pull deepseek-r1:14b # reasoning and complex tasks
ollama pull devstral # Mistral open coder
ollama pull phi4-mini # fast lightweight responses
ollama pull nomic-embed-text # embeddings for semantic search
ollama pull llava:7b # vision modelCloud/heavy candidates are intentionally not part of the default pull list: deepseek-v4-flash:cloud is Ollama Cloud, not local weights, and llama4:maverick is roughly 245GB in Ollama. Use /model-fleet before pulling or promoting either one.
# Python tools (username scan + typosquatting)
pip install maigret dnstwist
# Subdomain enumeration (passive, no API key required)
brew install subfinderAll four tools (maigret, dnstwist, subfinder, whois) are discovered at runtime via shutil.which — Jarvis reports availability at GET /osint/status.
./run.shHeadless:
./run.sh --no-uibash scripts/install_jarvis_app.shThat rebuilds the latest packaged app and installs it to both:
Applications/Jarvis.appDesktop/Jarvis.app
For the deeper system docs, start here:
- docs/jarvis_architecture/00_ARCHITECTURE.md
- docs/jarvis_architecture/02_MEMORY_ARCHITECTURE.md
- docs/jarvis_architecture/03_CAPABILITY_MODULES.md
- docs/jarvis_architecture/05_OPEN_SOURCE_FIRST_ROADMAP.md
- docs/jarvis_architecture/06_PROJECT_AUDIT_2026_04_09.md
- docs/jarvis_architecture/07_MEMPALACE_GRAPHIFY_ADOPTION.md
- docs/persistent-jarvis-v1.md
Webhook hardening env vars: JARVIS_WEBHOOK_MAX_AGE_SECONDS and JARVIS_PERSIST_REDACT_SOURCES (see docs/persistent-jarvis-v1.md).
If you remember only one thing, remember this:
Jarvis is not one model.
Jarvis is:
- a runtime
- a memory system
- a router
- a tool layer
- a desktop app
- an extension surface
- an always-on brain core
- a parallel agent dispatcher
- a 14-agent specialized task team with real, sandboxed tools
- a verification loop that checks what agents actually did, not what they claim
- a 3D office world that shows every agent working in real time
- and a roadmap toward a real local AI assistant
The model is just one part inside that machine.