Author: Prashant Devani

AI Computer Assistant

A production-quality AI agent dashboard that extends the Vercel AI SDK computer use demo. This application provides a robust, two-panel interface for interacting with an AI agent capable of controlling a remote computer environment via E2B.

📋 Task Overview

This project was built to transform a simple demo into a comprehensive agentic workflow tool. Key objectives achieved include:

1. Two-Panel Dashboard: A resizable split-screen layout featuring a chat interface on the left and a live VNC desktop viewer on the right.
2. Session Management: Full support for creating, switching, and deleting multiple chat sessions, with history persisted locally.
3. Event Pipeline: A typed state management system that captures tool calls, execution status, and durations for debugging and visualization.
4. Robust Architecture: Implementation of TypeScript best practices and a decoupled chat architecture to ensure state isolation between sessions.

🚀 Getting Started

Prerequisites

• Node.js: v20+
• Docker (optional, for containerized deployment)
• API Keys:
  • ANTHROPIC_API_KEY: For the AI model (Claude).
  • E2B_API_KEY: For the sandboxed computer environment.

Environment Setup

Create a .env.local file in the root directory:

ANTHROPIC_API_KEY=your_anthropic_key
E2B_API_KEY=your_e2b_key

# Optional UI Configuration
NEXT_PUBLIC_GITHUB_URL="https://github.com/your-repo"
NEXT_PUBLIC_GITHUB_BUTTON_LABEL="Code"
NEXT_PUBLIC_STORYBOOK_URL="/storybook/"
NEXT_PUBLIC_STORYBOOK_BUTTON_LABEL="Storybook"

Running with Docker (Recommended)

This project includes a production-ready Docker setup that builds both the Next.js application and the Storybook documentation.

1. Build and Run:

   docker-compose up --build

   This will automatically load variables from your .env.local file.

2. Access:

   • App: http://localhost:3000
   • Storybook: http://localhost:3000/storybook

Running Locally

1. Install Dependencies:

   npm install --legacy-peer-deps

2. Start Development Server:

   npm run dev

   Access the app at http://localhost:3000.

3. Run Storybook:

   npm run storybook

   Access the component library at http://localhost:6006.

✨ Features

🤖 Chatbot AI

The chat interface is powered by the Vercel AI SDK and Anthropic's Claude 3.5 Sonnet (or newer). It supports:

🖥️ Right Panel Overlay & Debug Log

A dynamic overlay system for the right panel that provides deep context without cluttering the UI. Supported views include:

• Debug Console (DEBUG_PANEL): Real-time JSON activity log of every agent action (tool calls, navigation, errors). Integrates @uiw/react-json-view for inspecting complex objects.
• Tool Details (TOOL_DETAILS): Deep dive into specific tool invocations (e.g., full bash output or API arguments).
• Screenshot Viewer (SCREENSHOT_VIEWER): Full-resolution inspection of agent-captured images.
• Event State (EVENT_STATE_VIEWER): Raw state inspection for debugging the event pipeline.

🤖 E2B Desktop Session Manager

Robust VM management integrated directly into the chat workflow:

• 1:1 Session Binding: Each chat session is uniquely bound to a dedicated E2B desktop instance.
• Auto-Recovery: If a connection drops or the 404s, the system automatically provisions a fresh VM and updates the session credentials.
• Smart Persistence: Desktop stream URLs and Sandbox IDs are persisted in Redux for seamless tab switching, but cleared on browser close to prevent stale connections ("Fresh Start" logic).

📱 Responsive Design

Fully optimized for all devices:

• Desktop/Tablet: Resizable split-pane layout (Chat Left / VNC Right).
• Mobile: Unified view with the VNC desktop front-and-center. Chat is accessible via a floating action button, and overlays (debug logs, tools) slide up smoothly over the interface.

🎨 Theme System

The application features a fully responsive Dark/Light mode system.

• Tailwind CSS: Used for all styling, ensuring consistency and ease of customization.
• Shadcn UI: Built on top of Radix UI primitives for accessible, high-quality components.
• CSS Variables: Colors are defined in globals.css using HSL variables, making global theme updates trivial.

🛠️ Technical Architecture

Data Persistence (Local Storage)

Application state is managed using Redux Toolkit.

• Middleware: A custom middleware listens to Redux actions and syncs specific slices (like chat sessions) to the browser's localStorage.
• Hydration: On app startup, state is rehydrated from local storage, allowing users to pick up exactly where they left off.

Chat System Architecture

To solve common issues with state leaks in single-page apps (SPA), the chat system was refactored into a ChatContainer architecture.

• Isolation: The <ChatContainer /> component renders the useChat hook.
• Key-based Remounting: The parent layout renders <ChatContainer key={activeSessionId} />. Changing the key forces React to completely unmount the old chat and mount a fresh instance.
• Benefit: This guarantees that when switching sessions, zero state leaks from the previous conversation. You always get a fresh message list (or the correct restored history) without "ghost" messages.

Storybook Implementation

We use Storybook to document and test components in isolation.

• Coverage: Stories exist for UI primitives (Button, Input) and complex feature components (ChatInput, ProjectInfo).
• Docker Integration: The specific Dockerfile builds the static Storybook site (npm run build-storybook) and serves it under the /storybook route in the production container.
• Routing: A next.config.ts redirect ensures visiting /storybook automatically serves the correct index.html.

🚀 Improvement Areas

While the current system is robust, future enhancements could include:

1. Backend Persistence: Move from localStorage to a proper database (PostgreSQL/Supabase) to support cross-device sync.
2. Mobile Optimization: Further refine the VNC viewer controls for touch devices.
3. Agent Customization: Allow users to configure the system prompt or choose different models via the UI.
4. Multi-Agent Support: Orchestrate multiple agents working in the same sandbox.

---

🔗 Resources

• Demo Video: https://drive.google.com/file/d/1EgsYwEnCpZoLpGLEXTqEDtlIz3iNTpr1/view?usp=sharing
• Live App (Alpha): https://ai-computer-assistant-alpha.labstack.site/

  Note: The live app is hosted on a homelab and may be unstable. For the best experience, please run the project locally using Docker.

---

Built using Next.js, AI SDK, and E2B.