GoScraper 🚀

Enterprise-Grade Web Scraping Library & Microservice for Go

Modern, fast, and stealth web scraping library with AI-powered extraction, anti-bot detection, and microservice architecture. Perfect for e-commerce, news, and data extraction at scale.

🌟 Key Features

🤖 AI-Powered Smart Extraction

Multiple AI Providers: OpenAI GPT-4, Anthropic Claude, Local models
Smart Content Detection: Automatically identifies and extracts structured data
Confidence Scoring: Quality assurance for extracted data
Fallback Chain: CSS/XPath extraction when AI fails

🏗️ Microservice Architecture

HTTP API Server: RESTful endpoints for scraping operations
Docker Support: Container-ready with Docker Compose
Kubernetes Ready: Production deployment manifests included
Load Balancing: Nginx configuration for horizontal scaling

⚙️ Flexible Configuration System

JSON Configuration: File-based configuration management
Environment Variables: 12-factor app compliance
CLI Tools: Interactive setup and validation
Hot Reloading: Runtime configuration updates

🌐 Multi-Engine Browser Support

ChromeDP: High-performance Chrome automation
Rod: Lightning-fast browser control
Stealth Mode: Advanced anti-detection techniques
Headless & GUI: Flexible rendering options

🚀 Production Features

Rate Limiting: Configurable request throttling
Caching: Redis and in-memory caching
Proxy Support: IP rotation and geo-targeting
Health Checks: Monitoring and observability
Graceful Shutdown: Clean resource management

📦 Installation

go get github.com/ramusaaa/goscraper

🚀 Quick Start

Method 1: Interactive Setup (Recommended)

# 1. Initialize configuration
make init-config

# 2. Interactive setup wizard
make setup
# Follow prompts to configure AI keys, caching, etc.

# 3. Validate configuration
make validate-config

# 4. Start the server
make run

Method 2: Environment Variables

# Set your API keys
export OPENAI_API_KEY="your-openai-key"
export GOSCRAPER_AI_ENABLED=true

# Start the server
go run ./cmd/api

Method 3: Manual Configuration

# Create config file
cp goscraper.example.json goscraper.json

# Edit configuration
vim goscraper.json

# Start server
go run ./cmd/api

💻 Usage Examples

Basic Library Usage

package main

import (
    "fmt"
    "log"
    
    "github.com/ramusaaa/goscraper"
)

func main() {
    // Simple scraping
    scraper := goscraper.New()
    
    resp, err := scraper.Get("https://example.com")
    if err != nil {
        log.Fatal(err)
    }
    
    title := resp.Document.Find("title").Text()
    fmt.Printf("Page title: %s\n", title)
}

Advanced Configuration

scraper := goscraper.New(
    goscraper.WithTimeout(30*time.Second),
    goscraper.WithUserAgent("MyBot/1.0"),
    goscraper.WithHeaders(map[string]string{
        "Accept-Language": "en-US,en;q=0.9",
    }),
    goscraper.WithRateLimit(500*time.Millisecond),
    goscraper.WithMaxRetries(3),
    goscraper.WithProxy("http://proxy.example.com:8080"),
    goscraper.WithStealth(true),
)

HTTP API Usage

# Health check
curl http://localhost:8080/health

# Get configuration
curl http://localhost:8080/config

# Scrape a website
curl -X POST http://localhost:8080/api/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Smart AI-powered scraping
curl -X POST http://localhost:8080/api/smart-scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shop.example.com/products"}'

Client SDK Usage

package main

import (
    "fmt"
    "log"
    
    "github.com/ramusaaa/goscraper/client"
)

func main() {
    // Create client for remote scraper service
    client := client.NewScraperClient("http://localhost:8080")
    
    // Health check
    if err := client.Health(); err != nil {
        log.Fatal("Service unavailable:", err)
    }
    
    // Scrape website
    data, err := client.Scrape("https://example.com")
    if err != nil {
        log.Fatal("Scraping failed:", err)
    }
    
    fmt.Printf("Title: %s\n", data.Title)
    fmt.Printf("Status: %d\n", data.StatusCode)
}

📋 Configuration Reference

Configuration File Structure

{
  "server": {
    "port": "8080",
    "host": "0.0.0.0",
    "read_timeout": "30s",
    "write_timeout": "30s"
  },
  "ai": {
    "enabled": true,
    "provider": "openai",
    "confidence_threshold": 0.8,
    "fallback_chain": ["openai", "css", "xpath"],
    "models": {
      "openai": {
        "api_key": "your-openai-key",
        "model": "gpt-4"
      },
      "anthropic": {
        "api_key": "your-anthropic-key",
        "model": "claude-3-sonnet-20240229"
      }
    }
  },
  "browser": {
    "engine": "chromedp",
    "headless": true,
    "stealth": true,
    "pool_size": 5
  },
  "cache": {
    "enabled": true,
    "type": "redis",
    "ttl": "1h",
    "redis": {
      "host": "localhost",
      "port": 6379
    }
  },
  "rate_limit": {
    "requests_per_second": 10,
    "delay": "100ms"
  }
}

Environment Variables

# Server Configuration
GOSCRAPER_PORT=8080
GOSCRAPER_HOST=0.0.0.0

# AI Configuration
GOSCRAPER_AI_ENABLED=true
GOSCRAPER_AI_PROVIDER=openai
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

# Browser Configuration
GOSCRAPER_BROWSER_ENGINE=chromedp
GOSCRAPER_BROWSER_HEADLESS=true
GOSCRAPER_BROWSER_STEALTH=true

# Cache Configuration
GOSCRAPER_CACHE_ENABLED=true
GOSCRAPER_CACHE_TYPE=redis
REDIS_HOST=localhost
REDIS_PORT=6379

# Rate Limiting
GOSCRAPER_RATE_LIMIT_RPS=10
GOSCRAPER_RATE_LIMIT_DELAY=100ms

🛠️ CLI Tools

Available Commands

# Configuration Management
make init-config          # Create default configuration
make setup                # Interactive setup wizard
make validate-config      # Validate configuration
make show-config          # Display current configuration

# Development
make build                # Build binaries
make run                  # Start API server
make test                 # Run tests

# Docker
make docker-build         # Build Docker image
make docker-compose-up    # Start with Docker Compose
make docker-compose-down  # Stop Docker services

# Kubernetes
make k8s-deploy          # Deploy to Kubernetes
make k8s-delete          # Remove from Kubernetes

CLI Usage Examples

# Initialize new project
goscraper init

# Interactive setup
goscraper setup

# Validate configuration
goscraper validate

# Show current configuration
goscraper config

🏗️ Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Load Balancer │    │   API Gateway   │    │  Web Dashboard  │
│     (Nginx)     │    │   (Optional)    │    │   (Optional)    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Scraper Node 1 │    │  Scraper Node 2 │    │  Scraper Node N │
│                 │    │                 │    │                 │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │HTTP API     │ │    │ │HTTP API     │ │    │ │HTTP API     │ │
│ │Server       │ │    │ │Server       │ │    │ │Server       │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │Browser Pool │ │    │ │Browser Pool │ │    │ │Browser Pool │ │
│ │+ AI Engine  │ │    │ │+ AI Engine  │ │    │ │+ AI Engine  │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
    ┌─────────────────────────────────────────────────────────┐
    │                Infrastructure Layer                      │
    │                                                         │
    │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
    │  │    Redis    │  │  Config     │  │   Proxy     │     │
    │  │   Cache     │  │  Storage    │  │  Rotation   │     │
    │  └─────────────┘  └─────────────┘  └─────────────┘     │
    │                                                         │
    │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
    │  │   OpenAI    │  │ Anthropic   │  │   Local     │     │
    │  │    API      │  │    API      │  │   Models    │     │
    │  └─────────────┘  └─────────────┘  └─────────────┘     │
    └─────────────────────────────────────────────────────────┘

🚀 Deployment Options

1. Standalone Binary

# Build and run
go build -o goscraper ./cmd/api
./goscraper

2. Docker Container

# Build image
docker build -t goscraper:latest .

# Run container
docker run -p 8080:8080 \
  -e OPENAI_API_KEY=your-key \
  -e GOSCRAPER_AI_ENABLED=true \
  goscraper:latest

3. Docker Compose

# Start services
docker-compose up -d

# View logs
docker-compose logs -f scraper-api

# Stop services
docker-compose down

4. Kubernetes

# Deploy to cluster
kubectl apply -f k8s/

# Scale deployment
kubectl scale deployment scraper-api --replicas=5

# Check status
kubectl get pods -l app=scraper-api

🎯 Use Cases & Examples

E-commerce Price Monitoring

// Monitor product prices
data, err := goscraper.SmartScrape("https://shop.example.com/product/123")
if err != nil {
    log.Fatal(err)
}

if data.ContentType == goscraper.ContentTypeEcommerce {
    for _, product := range data.Products {
        fmt.Printf("Product: %s\n", product.Name)
        fmt.Printf("Price: %s %s\n", product.Price, product.Currency)
        fmt.Printf("Rating: %.1f/5\n", product.Rating)
    }
}

News Aggregation

// Extract news articles
data, err := goscraper.SmartScrape("https://news.example.com/article/123")
if err != nil {
    log.Fatal(err)
}

if data.ContentType == goscraper.ContentTypeNews && data.Article != nil {
    fmt.Printf("Headline: %s\n", data.Article.Headline)
    fmt.Printf("Author: %s\n", data.Article.Author)
    fmt.Printf("Published: %s\n", data.Article.PublishDate)
    fmt.Printf("Content: %s\n", data.Article.Content)
}

Job Listings Scraping

// Extract job postings
data, err := goscraper.SmartScrape("https://jobs.example.com/posting/123")
if err != nil {
    log.Fatal(err)
}

if data.ContentType == goscraper.ContentTypeJob && data.JobListing != nil {
    fmt.Printf("Title: %s\n", data.JobListing.Title)
    fmt.Printf("Company: %s\n", data.JobListing.Company)
    fmt.Printf("Salary: %s\n", data.JobListing.Salary)
    fmt.Printf("Location: %s\n", data.JobListing.Location)
}

Microservice Integration

// Use as microservice client
client := client.NewScraperClient("http://scraper-service:8080")

// Health check
if err := client.Health(); err != nil {
    log.Fatal("Scraper service unavailable")
}

// Batch scraping
urls := []string{
    "https://site1.com",
    "https://site2.com", 
    "https://site3.com",
}

for _, url := range urls {
    data, err := client.Scrape(url)
    if err != nil {
        log.Printf("Failed to scrape %s: %v", url, err)
        continue
    }
    
    fmt.Printf("Scraped %s: %s\n", url, data.Title)
}

📊 API Reference

HTTP Endpoints

Endpoint	Method	Description	Example
`/health`	GET	Health check and status	`curl http://localhost:8080/health`
`/config`	GET	Current configuration	`curl http://localhost:8080/config`
`/api/scrape`	POST	Basic web scraping	See below
`/api/smart-scrape`	POST	AI-powered extraction	See below

API Examples

Basic Scraping

curl -X POST http://localhost:8080/api/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "options": {
      "timeout": "30s",
      "user_agent": "Custom Bot"
    }
  }'

Smart AI Scraping

curl -X POST http://localhost:8080/api/smart-scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example.com/product/123"
  }'

Response Format

{
  "success": true,
  "data": {
    "title": "Page Title",
    "description": "Meta description",
    "url": "https://example.com",
    "status_code": 200,
    "load_time": "1.234s",
    "html": "<!DOCTYPE html>..."
  }
}

🧪 Testing & Validation

Run Tests

# Run all tests
make test

# Run integration tests
go test ./tests/ -v

# Validate configuration
make validate-config

# Full feature validation
./validate_features.sh

Test Coverage

# Generate coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

📈 Performance & Monitoring

Performance Benchmarks

Feature	Performance	Scalability
HTTP Requests	1,000+ req/sec	Linear scaling
Browser Sessions	50+ concurrent	Auto-scaling
AI Extraction	10+ pages/sec	Model dependent
Cache Hit Ratio	90%+	Distributed
Memory Usage	<512MB	Configurable

Health Monitoring

# Check service health
curl http://localhost:8080/health

# Monitor with watch
watch -n 5 'curl -s http://localhost:8080/health | jq'

# Check configuration
curl http://localhost:8080/config | jq

Logging

# View logs in Docker
docker-compose logs -f scraper-api

# View logs in Kubernetes
kubectl logs -f deployment/scraper-api

# Custom log level
GOSCRAPER_LOG_LEVEL=debug go run ./cmd/api

🔧 Troubleshooting

Common Issues

Configuration Not Loading

# Check config file location
go run ./cmd/cli config

# Validate configuration
go run ./cmd/cli validate

# Reset to defaults
go run ./cmd/cli init

AI Features Not Working

# Check AI configuration
curl http://localhost:8080/config | jq '.data.ai_enabled'

# Verify API key
export OPENAI_API_KEY=your-key
go run ./cmd/cli validate

Performance Issues

# Check resource usage
docker stats scraper-api

# Monitor requests
curl http://localhost:8080/health

# Adjust rate limiting
export GOSCRAPER_RATE_LIMIT_RPS=5

Debug Mode

# Enable debug logging
export GOSCRAPER_LOG_LEVEL=debug

# Verbose output
go run ./cmd/api -v

# Profile performance
go run ./cmd/api -cpuprofile=cpu.prof

🤝 Contributing

Development Setup

# Clone repository
git clone https://github.com/ramusaaa/goscraper
cd goscraper

# Install dependencies
go mod tidy

# Run tests
make test

# Start development server
make run

Code Style

# Format code
go fmt ./...

# Lint code
golangci-lint run

# Generate documentation
godoc -http=:6060

Pull Request Process

Fork the repository
Create feature branch (git checkout -b feature/amazing-feature)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

ChromeDP - Browser automation
GoQuery - HTML parsing
Gorilla Mux - HTTP routing
Go-Redis - Redis client

🏆 Why Choose GoScraper?

Feature	GoScraper	Scrapy	Puppeteer	Selenium
Language	✅ Go	🐍 Python	🟨 JavaScript	🐍 Python/Java
Performance	✅ High	⚠️ Medium	⚠️ Medium	❌ Low
AI Integration	✅ Built-in	❌ External	❌ External	❌ External
Microservice Ready	✅ Native	⚠️ Custom	⚠️ Custom	❌ No
Configuration	✅ Advanced	⚠️ Basic	⚠️ Basic	⚠️ Basic
Stealth Features	✅ Advanced	⚠️ Limited	✅ Good	⚠️ Limited
Deployment	✅ Easy	⚠️ Medium	⚠️ Medium	❌ Complex

⭐ Star this repository if you find it useful!

🐛 Report Issues

💡 Request Features

💰 Become a Sponsor

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
.github		.github
client		client
cmd		cmd
config		config
docs		docs
examples		examples
internal		internal
k8s		k8s
pkg		pkg
tests		tests
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
client.go		client.go
config.go		config.go
detector.go		detector.go
doc.go		doc.go
docker-compose.yml		docker-compose.yml
extractors.go		extractors.go
go.mod		go.mod
go.sum		go.sum
goscraper.example.json		goscraper.example.json
goscraper.go		goscraper.go
helpers.go		helpers.go
nginx.conf		nginx.conf
parser.go		parser.go
presets.go		presets.go
scraper.go		scraper.go
validate_features.sh		validate_features.sh
version.go		version.go

Uh oh!

License

ramusaaa/goscraper

Folders and files

Latest commit

History

Repository files navigation

GoScraper 🚀

🌟 Key Features

🤖 AI-Powered Smart Extraction

🏗️ Microservice Architecture

⚙️ Flexible Configuration System

🌐 Multi-Engine Browser Support

🚀 Production Features

📦 Installation

🚀 Quick Start

Method 1: Interactive Setup (Recommended)

Method 2: Environment Variables

Method 3: Manual Configuration

💻 Usage Examples

Basic Library Usage

Advanced Configuration

HTTP API Usage

Client SDK Usage

📋 Configuration Reference

Configuration File Structure

Environment Variables

🛠️ CLI Tools

Available Commands

CLI Usage Examples

🏗️ Architecture Overview

🚀 Deployment Options

1. Standalone Binary

2. Docker Container

3. Docker Compose

4. Kubernetes

🎯 Use Cases & Examples

E-commerce Price Monitoring

News Aggregation

Job Listings Scraping

Microservice Integration

📊 API Reference

HTTP Endpoints

API Examples

Basic Scraping

Smart AI Scraping

Response Format

🧪 Testing & Validation

Run Tests

Test Coverage

📈 Performance & Monitoring

Performance Benchmarks

Health Monitoring

Logging

🔧 Troubleshooting

Common Issues

Configuration Not Loading

AI Features Not Working

Performance Issues

Debug Mode

🤝 Contributing

Development Setup

Code Style

Pull Request Process

📄 License

🙏 Acknowledgments

🏆 Why Choose GoScraper?

About

Resources

License

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 2

Sponsor this project

Uh oh!

Packages 0

Packages