OpenFront Scraper

Advanced data collection and analysis system for OpenFront.io lobby data with real-time scraping, game processing, and REST API.

🚀 Features

• Real-time lobby scraping - Collects lobby data every 5 seconds
• Intelligent game processing - Analyzes FFA games for spawn coordinates and winner data
• Automated API calls - Fetches detailed game data with rate limiting
• RESTful API - Serves current and historical data
• Database migrations - Schema versioning and updates
• Docker containerization - Portable and scalable deployment

📊 What it does

1. Scrapes current lobby data from OpenFront.io API
2. Extracts unique lobby IDs and metadata
3. Calls individual game APIs for detailed data
4. Processes FFA games to extract spawn coordinates and winner analysis
5. Serves data via REST API with historical views

🏗️ Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Job Scheduler │    │   Express API    │    │   PostgreSQL    │
│                 │    │                  │    │                 │
│ • Scrape lobbies│    │ • /health        │    │ • snapshots     │
│ • Extract IDs   │◄──►│ • /current       │◄──►│ • game_data     │
│ • Call APIs     │    │ • /stats         │    │ • unique_lobbies│
│ • Process games │    │ • /history       │    │ • migrations    │
└─────────────────┘    └──────────────────┘    └─────────────────┘

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Node.js 16+ (for local development)

Run with Docker

# Clone the repository
git clone <your-repo-url>
cd openfront-scraper

# Start the entire system
docker-compose up -d

# Check if everything is running
curl http://localhost:3000/health

Local Development

# Install dependencies
npm install

# Set up environment variables
cp .env.example .env
# Edit .env with your database settings

# Run database migrations
npm run migrate

# Start the API server (includes job scheduler)
npm run dev:watch

📡 API Endpoints

Endpoint	Description	Example
GET /health	System health check	{"status": "healthy", "version": "2.0.0"}
GET /current	Latest lobby snapshot	Current lobbies with timestamp
GET /stats?days=7	Statistics for last N days	Snapshot counts and breakdowns
GET /history/24	Snapshots from last N hours	Time-series lobby data

Example API Usage

# Get current lobby data
curl http://localhost:3000/current

# Get stats for last 7 days  
curl http://localhost:3000/stats?days=7

# Get last 24 hours of snapshots
curl http://localhost:3000/history/24

🔧 Configuration

Environment variables (create .env file):

# Database
DATABASE_URL=postgres://scraper:password@localhost:5432/openfront
DB_HOST=localhost
DB_NAME=openfront
DB_USER=scraper
DB_PASSWORD=password

# API
PORT=3000

# Scraping (optional)
SCRAPE_INTERVAL=5000
OPENFRONT_API_URL=https://openfront.io/api/public_lobbies

🗄️ Database Schema

The system uses PostgreSQL with the following main tables:

• snapshots - Raw lobby data collected every 5 seconds
• unique_lobbies - Extracted lobby metadata and processing status
• game_data - Processed FFA game analysis with spawn coordinates
• game_api_responses - Temporary storage for detailed game data
• schema_migrations - Database migration tracking

🔄 Job System

The system runs several automated jobs:

Job	Schedule	Purpose
scrape-lobbies	Every 5 seconds	Collect current lobby data
extract-lobby-ids	Every 30 seconds	Extract unique lobbies from snapshots
call-game-api	Every 15 minutes	Fetch detailed game data
process-ffa-games	Every 5 minutes	Analyze FFA games for spawn data

📈 Data Processing

FFA Game Analysis

For each Free-for-All game, the system:

1. Fetches detailed game data from OpenFront API
2. Analyzes turn-by-turn data to find spawn coordinates
3. Identifies winner and their spawn location
4. Stores processed data for analysis

Example Processed Data

{
  "game_id": "abc123",
  "game_map": "europe", 
  "winner_username": "player1",
  "spawn_x": 45,
  "spawn_y": 67,
  "start_time": "2024-01-01T12:00:00Z"
}

🛠️ Development

Running Migrations

# Run pending migrations
npm run migrate

# Reset a specific migration
npm run migrate:reset 001-cleanup-unused-tables

Adding New Jobs

1. Create job file in src/jobs/
2. Add to src/jobs/job-registry.js
3. Jobs automatically get scheduled and executed

Project Structure

├── src/
│   ├── jobs/              # All processing jobs
│   └── scheduler/         # Job scheduling system
├── api/                   # Express API server
├── shared/                # Database connections & utilities  
├── sql/
│   └── migrations/        # Database schema changes
├── scripts/               # Utility scripts
├── docker-compose.yml     # Container orchestration
└── Dockerfile            # Container definition

📊 Monitoring

Health Checks

# Basic health check
curl http://localhost:3000/health

# Database connectivity
docker-compose exec app npm run test

Logs

# View real-time logs
docker-compose logs -f app

# View specific service logs
docker-compose logs postgres

🚢 Deployment

Docker Production

# Build and deploy
docker-compose -f docker-compose.prod.yml up -d

# Scale processing
docker-compose -f docker-compose.prod.yml up -d --scale processor=3

Environment Setup

1. Set production environment variables
2. Configure database with proper credentials
3. Set up monitoring and alerting
4. Configure log aggregation

📈 Roadmap

• v1.0 - Basic scraper
• v1.1 - Database storage
• v1.2 - Automatic scraping
• v2.0 - Docker deployment
• v3.0 - Advanced game processing
• v4.0 - Team game analysis
• v4.1 - Player statistics and rankings
• v4.2 - Map-specific spawn analysis
• v5.0 - Real-time dashboard
• v5.1 - API rate limiting and authentication

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests and ensure migrations work
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

• OpenFront.io for providing the game data API
• PostgreSQL community for the excellent database
• Docker for containerization platform