Feat/twilio devtools

.env.example

@@ -0,0 +1,17 @@
+# Twilio Network Traversal Service Credentials
+# Get these from your Twilio Console:
+# 1. Go to https://console.twilio.com/
+# 2. Navigate to Account > API Keys & Tokens
+# 3. Create a new API Key
+# 4. Use the SID as TWILIO_ACCOUNT_SID
+# 5. Use the Secret as TWILIO_AUTH_TOKEN
+TWILIO_ACCOUNT_SID=SK...your_api_key_sid_here
+TWILIO_AUTH_TOKEN=your_api_key_secret_here
+
+# Google Cloud Configuration
+# If not provided, will use current gcloud config
+PROJECT_ID=your-gcp-project-id
+# REGION=us-central1
+
+# Optional: Service Configuration
+# SERVICE_NAME=kernel-browser

.gitignore

@@ -0,0 +1,54 @@
+# Environment variables
+.env
+.env.local
+*.env
+!.env.example
+
+# Node modules
+node_modules/
+
+# Build outputs
+dist/
+build/
+out/
+
+# Logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# IDE files
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Temporary files
+tmp/
+temp/
+*.tmp
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+.Python
+venv/
+env/
+
+# Google Cloud
+.gcloudignore
+gcs-key.json
+service-account-key.json
+
+# Docker
+.dockerignore
+
+# Backup files
+*.bak
+*.backup

DEPLOYMENT.md

@@ -0,0 +1,166 @@
+# Kernel Browser - Cloud Run Deployment Guide
+
+This guide explains how to deploy the Kernel Browser to Google Cloud Run with secure Twilio credential management.
+
+## Prerequisites
+
+- Google Cloud SDK (`gcloud`) installed
+- Docker installed
+- Git installed
+- A Google Cloud Project with billing enabled
+- Twilio account with API credentials (for WebRTC TURN servers)
+
+## Quick Start
+
+### 1. Clone the repository
+```bash
+git clone <repository-url>
+cd browser-web-agent
+git submodule update --init --recursive
+```
+
+### 2. Set up Twilio credentials
+```bash
+# Copy the example environment file
+cp .env.example .env
+
+# Edit .env and add your Twilio credentials
+# Get these from https://console.twilio.com/ > Account > API Keys & Tokens
+```
+
+Your `.env` file should contain:
+```
+TWILIO_ACCOUNT_SID=SK...your_api_key_sid_here
+TWILIO_AUTH_TOKEN=your_api_key_secret_here
+```
+
+### 3. Deploy to Cloud Run
+```bash
+./deploy.sh
+```
+
+The script will:
+- Load credentials from `.env`
+- Create/update secrets in Google Secret Manager
+- Build and deploy the container to Cloud Run
+- Configure all necessary permissions
+
+## Deployment Options
+
+### Using Cloud Build (recommended)
+```bash
+./deploy.sh
+```
+
+### Using local Docker build
+```bash
+./deploy.sh --local
+```
+
+### Specify project and region
+```bash
+./deploy.sh --project YOUR_PROJECT_ID --region us-central1
+```
+
+## How It Works
+
+### Credential Management
+
+1. **Local Development**: Credentials are stored in `.env` file (gitignored)
+2. **Secret Manager**: Deploy script automatically creates/updates secrets in Google Secret Manager
+3. **Cloud Run**: Service uses `secretKeyRef` to securely access credentials at runtime
+4. **Dynamic TURN**: Container fetches fresh TURN credentials from Twilio on startup
+
+### Security Features
+
+- Credentials never appear in code or logs
+- Secrets are encrypted at rest and in transit
+- Service account has minimal required permissions
+- Automatic credential rotation support
+
+### Files Overview
+
+- `.env.example` - Template for environment variables
+- `.env` - Your local credentials (gitignored)
+- `deploy.sh` - Main deployment script with Secret Manager integration
+- `service-secrets.yaml` - Cloud Run config with secret references
+- `service.yaml` - Fallback config (for deployments without secrets)
+- `cloudbuild.yaml` - Cloud Build configuration
+- `twilio/` - Twilio credential management scripts
+
+## Updating Credentials
+
+To update Twilio credentials:
+
+1. Update `.env` with new credentials
+2. Run `./deploy.sh` again
+3. Script will update secrets and redeploy
+
+## Manual Secret Management
+
+If you need to manage secrets manually:
+
+```bash
+# Create secrets
+echo -n "YOUR_SID" | gcloud secrets create twilio-account-sid --data-file=-
+echo -n "YOUR_TOKEN" | gcloud secrets create twilio-auth-token --data-file=-
+
+# Update secrets
+echo -n "NEW_SID" | gcloud secrets versions add twilio-account-sid --data-file=-
+echo -n "NEW_TOKEN" | gcloud secrets versions add twilio-auth-token --data-file=-
+
+# Grant access to service account
+gcloud secrets add-iam-policy-binding twilio-account-sid \
+  --member="serviceAccount:kernel-browser-sa@PROJECT_ID.iam.gserviceaccount.com" \
+  --role="roles/secretmanager.secretAccessor"
+```
+
+## Service Endpoints
+
+After deployment, you'll have access to:
+
+- **Main Interface**: `https://SERVICE_URL/`
+- **WebRTC Client**: `https://SERVICE_URL/`
+- **Chrome DevTools**: `https://SERVICE_URL/devtools/`
+- **DevTools WebSocket**: `wss://SERVICE_URL/cdp/ws`
+- **Recording API**: `https://SERVICE_URL/api`
+- **Health Check**: `https://SERVICE_URL/health`
+
+## Troubleshooting
+
+### Deployment fails
+- Check that all prerequisites are installed
+- Ensure billing is enabled on your GCP project
+- Verify you have sufficient quota in your region
+
+### WebRTC not working
+- Ensure Twilio credentials are correct
+- Check Cloud Run logs: `gcloud run services logs read kernel-browser --region=us-central1`
+- Verify TURN servers are accessible from your network
+
+### Secrets not found
+- Run `gcloud secrets list` to verify secrets exist
+- Check service account permissions
+- Ensure Secret Manager API is enabled
+
+## Architecture
+
+```
+┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
+│   Client    │────▶│   Cloud Run      │────▶│ Secret Manager  │
+│  (Browser)  │     │   (Container)    │     │  (Credentials)  │
+└─────────────┘     └──────────────────┘     └─────────────────┘
+                            │
+                            ▼
+                    ┌──────────────────┐
+                    │   Twilio API     │
+                    │  (TURN Servers)  │
+                    └──────────────────┘
+```
+
+## Support
+
+For issues or questions:
+- Check logs: `gcloud run services logs read kernel-browser --region=us-central1`
+- Review service status: `gcloud run services describe kernel-browser --region=us-central1`
+- File an issue on GitHub

Dockerfile.cloudrun

@@ -1,3 +1,57 @@
+# DevTools Frontend build stage using browser-operator-core
+FROM --platform=linux/amd64 ubuntu:22.04 AS devtools-builder
+
+# Cache bust argument to force rebuilds
+ARG CACHE_BUST
+
+# Install required packages for DevTools frontend build
+RUN apt-get update && apt-get install -y \
+    curl \
+    git \
+    python3 \
+    python3-pip \
+    python-is-python3 \
+    wget \
+    unzip \
+    sudo \
+    ca-certificates \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Node.js 18.x
+RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
+    apt-get install -y nodejs && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace
+
+# Clone depot_tools
+RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+ENV PATH="/workspace/depot_tools:${PATH}"
+ENV DEPOT_TOOLS_UPDATE=0
+
+# Follow README instructions exactly - fetching code
+RUN mkdir devtools
+WORKDIR /workspace/devtools
+RUN fetch devtools-frontend
+
+# Build steps
+WORKDIR /workspace/devtools/devtools-frontend
+
+RUN gclient sync
+RUN /workspace/depot_tools/ensure_bootstrap
+
+# Build standard DevTools first
+RUN npm run build
+
+# Add Browser Operator fork and switch to it
+RUN git remote add upstream https://github.com/BrowserOperator/browser-operator-core.git
+RUN git fetch upstream
+RUN git checkout upstream/main
+
+# Build Browser Operator version
+RUN npm run build
+
 # Multi-stage build using kernel-images as base
 FROM docker.io/golang:1.25.0 AS server-builder
 WORKDIR /workspace/server
@@ -90,6 +144,12 @@ RUN apt-get update && \
     nginx \
     # PPA req
     software-properties-common && \
+    # Disable nginx auto-start to prevent conflicts with custom config
+    systemctl disable nginx || true && \
+    systemctl mask nginx || true && \
+    # Remove default nginx config to prevent conflicts
+    rm -f /etc/nginx/sites-enabled/default && \
+    rm -f /etc/nginx/nginx.conf && \
     # Userland apps
     sudo add-apt-repository ppa:mozillateam/ppa && \
     sudo apt-get install -y --no-install-recommends \
@@ -186,19 +246,40 @@ COPY kernel-images/images/chromium-headful/supervisor/services/ /etc/supervisor/
 # Copy the kernel-images API binary
 COPY --from=server-builder /out/kernel-images-api /usr/local/bin/kernel-images-api
 
-# Cloud Run specific: nginx configuration for port proxying
-COPY nginx.conf /etc/nginx/nginx.conf
+# ============================================================================
+# DevTools Integration
+# ============================================================================
+
+# Copy DevTools static files from builder
+COPY --from=devtools-builder /workspace/devtools/devtools-frontend/out/Default/gen/front_end /usr/share/nginx/devtools
+
+# Set permissions for DevTools files
+RUN chown -R kernel:kernel /usr/share/nginx/devtools
+
+# Cloud Run specific: wrapper scripts (nginx config is inline)
+# DO NOT copy nginx.conf to avoid auto-start conflicts
 COPY cloudrun-wrapper.sh /cloudrun-wrapper.sh
-RUN chmod +x /cloudrun-wrapper.sh
+COPY twilio/twilio-credential-updater.sh /twilio-credential-updater.sh
+RUN chmod +x /cloudrun-wrapper.sh /twilio-credential-updater.sh
+
+# Add essential services for neko WebRTC and Chromium
+COPY supervisor/services-cloudrun/dbus.conf /etc/supervisor/conf.d/services-cloudrun/dbus.conf
+COPY supervisor/services-cloudrun/xorg.conf /etc/supervisor/conf.d/services-cloudrun/xorg.conf
+COPY supervisor/services-cloudrun/neko.conf /etc/supervisor/conf.d/services-cloudrun/neko.conf
+COPY supervisor/services-cloudrun/chromium.conf /etc/supervisor/conf.d/services-cloudrun/chromium.conf
+COPY supervisor/services-cloudrun/devtools-frontend.conf /etc/supervisor/conf.d/services-cloudrun/devtools-frontend.conf
 
 # Create nginx temp directories for non-root execution
 RUN mkdir -p /tmp/nginx_client_temp /tmp/nginx_proxy_temp /tmp/nginx_fastcgi_temp \
-    /tmp/nginx_uwsgi_temp /tmp/nginx_scgi_temp && \
+    /tmp/nginx_uwsgi_temp /tmp/nginx_scgi_temp \
+    /tmp/nginx_devtools_client_temp /tmp/nginx_devtools_proxy_temp /tmp/nginx_devtools_fastcgi_temp \
+    /tmp/nginx_devtools_uwsgi_temp /tmp/nginx_devtools_scgi_temp && \
     chown -R kernel:kernel /tmp/nginx_*
 
 # Create supervisor log directories
 RUN mkdir -p /var/log/supervisord/chromium /var/log/supervisord/neko /var/log/supervisord/xorg \
-    /var/log/supervisord/dbus /var/log/supervisord/kernel-images-api /var/log/supervisord/mutter && \
+    /var/log/supervisord/dbus /var/log/supervisord/kernel-images-api /var/log/supervisord/mutter \
+    /var/log/supervisord/nginx /var/log/supervisord/devtools-frontend && \
     chown -R kernel:kernel /var/log/supervisord
 
 # Create health check endpoint