Framework - AI-Powered Browser Automation SDK

A TypeScript Electron application built with Bun, featuring an intelligent browser automation SDK powered by Gemini 2.5 Flash AI.

🚀 Features

• 🤖 AI Agent: Gemini-powered intelligent automation that understands natural language commands
• 🌐 Enhanced Browser: Playwright wrapper with advanced features and configurations
• 🔍 Smart DOM Analysis: Intelligent DOM tree parsing and element identification
• 🖥️ Electron Desktop App: Modern cross-platform desktop interface for automation control
• 📊 Real-time Progress: Live task execution monitoring and progress updates
• 🎯 Natural Language: Execute complex automation tasks with simple English commands

📁 Project Structure

framework/
├── src/
│   ├── agent/           # AI Agent with Gemini integration
│   │   └── Agent.ts     # Intelligent automation agent
│   ├── browser/         # Enhanced Browser automation
│   │   └── Browser.ts   # Playwright wrapper with advanced features
│   ├── dom/             # DOM analysis and parsing
│   │   └── DomTree.ts   # Smart DOM tree manipulation
│   ├── main.ts          # Main Electron process
│   ├── preload.ts       # Preload script for secure IPC
│   └── index.ts         # SDK exports
├── renderer/
│   └── index.html       # Modern automation control UI
├── recordings/          # Browser session recordings
├── screenshots/         # Automated screenshots
├── dist/                # Built application files
└── package.json         # Project configuration

🛠️ Setup

Prerequisites

• Bun >= 1.0 (recommended)
• Node.js >= 18 (fallback)
• TypeScript >= 5.0
• Gemini API Key from Google AI Studio

Installation

# Install dependencies
bun install

# Install Playwright browsers
bunx playwright install

# Create environment file
echo "GEMINI_API_KEY=your_gemini_api_key_here" > .env

Environment Variables

IMPORTANT: You need a real Gemini API key for the automation to work.

1. Get your API key from Google AI Studio
2. Create the .env file:

   echo "GEMINI_API_KEY=your_actual_api_key_here" > .env

Quick Test

Before using the UI, test that everything is working:

# Run the comprehensive test
bun quick-test.js

This will verify:

• ✅ API key is set correctly
• ✅ Browser automation works
• ✅ AI agent can execute tasks

If the test passes, you're ready to use the full application!

Scripts

# Development - Build and run the app
bun run dev

# Build the application
bun run build:all

# Run the built application
bun run start

# Development with hot reload
bun run electron:dev

# Type checking
bun run typecheck

# Clean build artifacts
bun run clean

📦 Building for Distribution

Build for current platform

bun run dist

Build for specific platforms

# macOS
bun run dist:mac

# Windows
bun run dist:win

# Linux
bun run dist:linux

Built applications will be available in the release/ directory.

🏗️ Architecture

🤖 AI Agent (src/agent/Agent.ts)

• Gemini 2.5 Flash Integration: Natural language understanding and decision making
• Task Planning: Iterative step-by-step automation execution
• DOM Analysis: Intelligent page structure understanding
• Action Execution: Browser automation with real-time feedback

🌐 Enhanced Browser (src/browser/Browser.ts)

• Playwright Wrapper: Advanced browser automation capabilities
• Multi-browser Support: Chromium, Firefox, and WebKit
• Recording & Screenshots: Built-in session recording and error capture
• Performance Monitoring: Page metrics and network analysis

🔍 DOM Tree Analysis (src/dom/DomTree.ts)

• Smart Parsing: Intelligent HTML structure analysis
• Element Detection: Automatic form, button, and link identification
• Content Extraction: Main content and navigation recognition
• Selector Generation: Robust CSS selector creation

🖥️ Electron Interface

• Main Process: Manages automation tasks and browser instances
• Renderer Process: Modern UI for task input and progress monitoring
• Preload Script: Secure IPC communication bridge

🔧 Configuration

The application supports various configuration options through the Config interface:

interface Config {
  windowOptions?: {
    width?: number;
    height?: number;
    resizable?: boolean;
  };
}

🚦 Usage

Starting the Application

1. Launch the Automation UI:

   bun run dev

2. Use Natural Language Commands:

   • Type in the input field: "Go to google.com and search for cats"
   • Press Enter and watch the AI agent work!

Example Automation Commands

• "Navigate to github.com and find trending repositories"
• "Go to amazon.com and search for MacBook Pro"
• "Visit linkedin.com and check my notifications"
• "Open reddit.com and browse the front page"
• "Go to wikipedia.org and search for artificial intelligence"

Programmatic SDK Usage

import { Agent, Browser, DomTree, createAgent } from './src/index.js';

// Quick start with default settings
const agent = createAgent({
  agentConfig: {
    debugMode: true,
    maxSteps: 15,
    stepDelay: 2000
  },
  browserConfig: {
    headless: false,
    timeout: 30000
  }
});

// Execute automation task
const result = await agent.executeTask("Go to google.com and search for TypeScript tutorials");
console.log('Task completed:', result);

// Advanced usage with individual components
const browser = new Browser({
  headless: false,
  viewport: { width: 1920, height: 1080 },
  recordVideo: true
});

await browser.launch();
await browser.goto('https://example.com');

const html = await browser.getContent();
const domTree = new DomTree(html, 'https://example.com');
const pageStructure = domTree.getPageStructure();

console.log('Interactive elements:', pageStructure.interactiveElements);
await browser.close();

🔒 Security

This application follows Electron security best practices:

• Context Isolation: Enabled for all renderer processes
• Node Integration: Disabled in renderer processes
• Preload Scripts: Used for secure IPC communication
• CSP Headers: Content Security Policy implemented
• Window Creation: Restricted and controlled

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📝 License

MIT License - see LICENSE file for details.

---

Built with ❤️ using Bun and Electron