Skip to content

Sekinal/aa_contest

BranchesTags

Repository files navigation

🛫 AA Flight Scraper

Production-ready American Airlines flight scraper with advanced bot evasion, automatic cookie management, and high-performance bulk scraping.

Docker Hub Python 3.12 License

Scrapes both Award and Revenue flights from American Airlines with automatic cookie extraction, HTTP/2 support, circuit breaker pattern, and adaptive rate limiting. Now with concurrent bulk scraping for multiple routes and dates!

🚀 Quick Start (Docker - Recommended)

The easiest way to use this scraper is via Docker. No local setup required!

Single Route Search

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15

🔥 NEW: Bulk Concurrent Scraping

Scrape multiple destinations and dates simultaneously!

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO \
  --destinations JFK MIA ORD DFW \
  --dates 2025-12-15 2025-12-16 2025-12-17 \
  --max-concurrent 5

This creates 2 × 4 × 3 = 24 route/date combinations that run concurrently! 🚀

Key benefits:

  • ⚡ 10-20x faster than sequential scraping
  • 🔄 Shared cookie management across all tasks
  • 🛡️ Built-in rate limiting and error isolation
  • 📊 Progress tracking for each combination
  • 🎯 One failed route doesn't stop others

That's it! The scraper will:

  1. ✅ Automatically extract cookies using a real browser (Camoufox/Firefox)
  2. ✅ Search for both Award and Revenue flights across all combinations
  3. ✅ Save results to ./output/
  4. ✅ Log everything to ./logs/
  5. ✅ Cache cookies in ./cookies/ for reuse

Example Output

================================================================================
🚀 BULK CONCURRENT SCRAPING MODE
================================================================================
Origins:       LAX, SFO
Destinations:  JFK, MIA, ORD, DFW
Dates:         2025-12-15, 2025-12-16, 2025-12-17
Total combos:  24
Max concurrent: 10
Search types:  Award, Revenue
================================================================================

🔍 Starting: LAX → JFK on 2025-12-15
🔍 Starting: LAX → MIA on 2025-12-15
...
✅ Completed: LAX → JFK on 2025-12-15 (2/2 searches, 45 flights)
✅ Completed: SFO → ORD on 2025-12-16 (2/2 searches, 38 flights)
...

================================================================================
✅ BULK SCRAPING COMPLETE
================================================================================
Total combinations: 24
Successful:        23
Failed:            1
Duration:          45.2s
Avg per combo:     1.9s
================================================================================

📋 Docker Usage

Basic Single Search

Search for flights on a specific route and date:

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin SFO \
  --destination LAX \
  --date 2025-12-20

Bulk Scraping Options

Multiple Destinations, Single Date

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destinations JFK MIA ORD DFW ATL \
  --date 2025-12-15 \
  --max-concurrent 5

Multiple Dates, Single Destination

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --dates 2025-12-15 2025-12-16 2025-12-17 2025-12-18 2025-12-19 \
  --max-concurrent 5

Full Matrix: Multiple Origins, Destinations, and Dates

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO SEA \
  --destinations JFK BOS PHL DCA \
  --dates 2025-12-15 2025-12-20 2025-12-25 \
  --max-concurrent 5

I recommend that instead of increasing the concurrency, you instead increase the number of cookies extracted, simulating the requests of three different browsers! Much safer from my tests and less likely to get blocked, whilst still achieving high net concurrency.

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO SEA \
  --destinations JFK BOS PHL DCA \
  --dates 2025-12-15 2025-12-20 2025-12-25 \
  --max-concurrent 5 \
  --browsers 3

This creates 3 × 4 × 3 = 36 combinations!

Performance Tuning

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 5 \
  --verbose

Performance recommendations:

  • Safest: --max-concurrent 5 (This is the default option, totally safe)
  • Conservative: --max-concurrent 5 + --browsers 3 (pretty safe, haven't tested scraping continuously for more than 5 minutes though)
  • Balanced: --max-concurrent 10 + --browsers 3 (risky, likely to get blocked after a few minutes)
  • Aggressive: --max-concurrent 15 + --browsers 3 (fast, will likely be blocked if scraping for more than 1 minute)

Advanced Options

Specify Cabin Class

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin JFK \
  --destination LHR \
  --date 2025-12-25 \
  --cabin BUSINESS

Available cabin classes: COACH, BUSINESS, FIRST, PREMIUM_ECONOMY

Multiple Passengers

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin ORD \
  --destination DFW \
  --date 2025-12-15 \
  --passengers 2

Search Type (Award Only or Revenue Only)

We can only search one of the API endpoints, I don't recommend doing so as that defeats the purpose of the data processing pipeline, but you can do so:

# Award flights only
docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --search-type Award

# Revenue flights only
docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --search-type Revenue

Visible Browser (for debugging)

Watch the cookie extraction process in real-time:

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --no-headless

Verbose Logging

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --verbose

📂 Output Files

After running, you'll find:

output/
├── LAX_JFK_2025-12-15_20251106_010602_combined.json    # ✅ Merged Award + Revenue data
├── SFO_MIA_2025-12-16_20251106_010605_combined.json    # Another route/date combo
└── raw_data/
    ├── LAX_JFK_2025-12-15_20251106_010602_award_raw.json    # Raw Award API response
    ├── LAX_JFK_2025-12-15_20251106_010602_revenue_raw.json  # Raw Revenue API response
    ├── SFO_MIA_2025-12-16_20251106_010605_award_raw.json
    └── SFO_MIA_2025-12-16_20251106_010605_revenue_raw.json

cookies/
├── aa_cookies.json          # Extracted cookies (auto-cached)
├── aa_cookies_headers.json  # Request headers
└── aa_cookies_referer.txt   # Referer URL

logs/
└── aa_scraper.log           # Detailed logs with timestamps

Combined Output Format

The *_combined.json file contains merged Award + Revenue data:

{
  "search_metadata": {
    "origin": "LAX",
    "destination": "JFK",
    "date": "2025-12-15",
    "passengers": 1,
    "cabin_class": "economy"
  },
  "flights": [
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA28",
          "departure_time": "00:15",
          "arrival_time": "08:29"
        }
      ],
      "total_duration": "5h 14m",
      "points_required": 18000,
      "cash_price_usd": 208.48,
      "taxes_fees_usd": 5.6,
      "cpp": 1.13
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA118",
          "departure_time": "06:05",
          "arrival_time": "14:10"
        }
      ],
      "total_duration": "5h 5m",
      "points_required": 15000,
      "cash_price_usd": 208.48,
      "taxes_fees_usd": 5.6,
      "cpp": 1.35
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA2",
          "departure_time": "07:00",
          "arrival_time": "15:32"
        }
      ],
      "total_duration": "5h 32m",
      "points_required": 17500,
      "cash_price_usd": 208.48,
      "taxes_fees_usd": 5.6,
      "cpp": 1.16
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA307",
          "departure_time": "08:00",
          "arrival_time": "16:28"
        }
      ],
      "total_duration": "5h 28m",
      "points_required": 20000,
      "cash_price_usd": 258.49,
      "taxes_fees_usd": 5.6,
      "cpp": 1.26
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA238",
          "departure_time": "10:15",
          "arrival_time": "18:42"
        }
      ],
      "total_duration": "5h 27m",
      "points_required": 31000,
      "cash_price_usd": 368.48,
      "taxes_fees_usd": 5.6,
      "cpp": 1.17
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA32",
          "departure_time": "11:20",
          "arrival_time": "19:45"
        }
      ],
      "total_duration": "5h 25m",
      "points_required": 27000,
      "cash_price_usd": 298.49,
      "taxes_fees_usd": 5.6,
      "cpp": 1.08
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA274",
          "departure_time": "12:37",
          "arrival_time": "21:00"
        }
      ],
      "total_duration": "5h 23m",
      "points_required": 27500,
      "cash_price_usd": 368.48,
      "taxes_fees_usd": 5.6,
      "cpp": 1.32
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA4",
          "departure_time": "15:40",
          "arrival_time": "23:49"
        }
      ],
      "total_duration": "5h 9m",
      "points_required": 15000,
      "cash_price_usd": 143.48,
      "taxes_fees_usd": 5.6,
      "cpp": 0.92
    },
    {
      "is_nonstop": true,
      "segments": [
        {
          "flight_number": "AA10",
          "departure_time": "21:45",
          "arrival_time": "06:00"
        }
      ],
      "total_duration": "5h 15m",
      "points_required": 20000,
      "cash_price_usd": 258.49,
      "taxes_fees_usd": 5.6,
      "cpp": 1.26
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA1956",
          "departure_time": "00:45",
          "arrival_time": "06:46"
        },
        {
          "flight_number": "AA2848",
          "departure_time": "08:20",
          "arrival_time": "11:37"
        }
      ],
      "total_duration": "7h 52m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA1956",
          "departure_time": "00:45",
          "arrival_time": "06:46"
        },
        {
          "flight_number": "AA961",
          "departure_time": "10:06",
          "arrival_time": "13:29"
        }
      ],
      "total_duration": "9h 44m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA2808",
          "departure_time": "06:06",
          "arrival_time": "13:46"
        },
        {
          "flight_number": "AA3190",
          "departure_time": "17:05",
          "arrival_time": "19:00"
        }
      ],
      "total_duration": "9h 54m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA2068",
          "departure_time": "06:08",
          "arrival_time": "11:19"
        },
        {
          "flight_number": "AA1654",
          "departure_time": "12:39",
          "arrival_time": "17:15"
        }
      ],
      "total_duration": "8h 7m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA2497",
          "departure_time": "07:59",
          "arrival_time": "15:55"
        },
        {
          "flight_number": "AA4781",
          "departure_time": "17:54",
          "arrival_time": "19:17"
        }
      ],
      "total_duration": "8h 18m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA3176",
          "departure_time": "18:50",
          "arrival_time": "20:23"
        },
        {
          "flight_number": "AA276",
          "departure_time": "22:37",
          "arrival_time": "07:00"
        }
      ],
      "total_duration": "9h 10m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA6260",
          "departure_time": "20:00",
          "arrival_time": "21:30"
        },
        {
          "flight_number": "AA276",
          "departure_time": "22:37",
          "arrival_time": "07:00"
        }
      ],
      "total_duration": "8h 0m",
      "points_required": 19000,
      "cash_price_usd": 233.18,
      "taxes_fees_usd": 5.6,
      "cpp": 1.2
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA820",
          "departure_time": "22:37",
          "arrival_time": "06:12"
        },
        {
          "flight_number": "AA3199",
          "departure_time": "08:09",
          "arrival_time": "10:00"
        }
      ],
      "total_duration": "8h 23m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA2715",
          "departure_time": "23:59",
          "arrival_time": "04:58"
        },
        {
          "flight_number": "AA1554",
          "departure_time": "07:01",
          "arrival_time": "11:28"
        }
      ],
      "total_duration": "8h 29m",
      "points_required": 15000,
      "cash_price_usd": 226.19,
      "taxes_fees_usd": 5.6,
      "cpp": 1.47
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA6371",
          "departure_time": "17:03",
          "arrival_time": "18:30"
        },
        {
          "flight_number": "AA276",
          "departure_time": "22:37",
          "arrival_time": "07:00"
        }
      ],
      "total_duration": "10h 57m",
      "points_required": 20500,
      "cash_price_usd": 327.97,
      "taxes_fees_usd": 11.2,
      "cpp": 1.55
    },
    {
      "is_nonstop": false,
      "segments": [
        {
          "flight_number": "AA2453",
          "departure_time": "22:49",
          "arrival_time": "07:14"
        },
        {
          "flight_number": "AA4721",
          "departure_time": "12:03",
          "arrival_time": "13:29"
        }
      ],
      "total_duration": "11h 40m",
      "points_required": 15000,
      "cash_price_usd": 231.97,
      "taxes_fees_usd": 11.2,
      "cpp": 1.47
    }
  ],
  "total_results": 20
}

Key Fields:

  • points_required: Award miles needed
  • cash_price_usd: Revenue ticket price
  • cpp: Cents per point (award value calculation)
  • is_nonstop: Whether it's a direct flight
  • segments: Individual flight legs with times

🔧 Local Development (uv)

Want to develop locally or run without Docker? Use uv - the modern Python package manager.

Prerequisites

  • Python 3.12
  • uv (install: curl -LsSf https://astral.sh/uv/install.sh | sh)

Installation

# Clone the repository
git clone https://github.com/yourusername/aa-scraper.git
cd aa-scraper

# Install dependencies with uv
uv sync

This creates a virtual environment and installs all dependencies (including Camoufox browser).

Run Locally

# Single route
uv run -m aa_scraper \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15

# Bulk scraping
uv run -m aa_scraper \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 5

Development Commands

# Install dev dependencies
uv sync --all-extras

# Run tests
uv run pytest

# Format code
uv run black aa_scraper/

# Lint
uv run ruff check aa_scraper/

# Build Docker image locally
docker build -t aa-scraper:local .

🎯 Features

🚀 High-Performance Bulk Scraping

  • Concurrent Execution: Scrape 10-15+ routes simultaneously
  • Matrix Combinations: Origins × Destinations × Dates
  • Shared Cookie Pool: All tasks use same authentication
  • Error Isolation: One failed route doesn't stop others
  • Progress Tracking: Real-time status for each combination
  • Smart Semaphore: Configurable concurrency limits

🤖 Advanced Bot Evasion

  • Camoufox Browser: Uses Firefox-based browser with anti-detection
  • Automatic Cookie Extraction: Real browser session with proper headers
  • Akamai Bypass: Handles and solves Akamai bot challenges automatically
  • HTTP/2 Support: Native HTTP/2 with fallback to HTTP/1.1
  • Smart Header Ordering: Mimics real browser request patterns

🔄 Production-Ready Reliability

  • Circuit Breaker Pattern: Prevents cascading failures
  • Exponential Backoff: Smart retry logic with jitter
  • Rate Limiting: Adaptive rate limiter (1 req/s default)
  • Cookie Auto-Refresh: Detects expired cookies and refreshes automatically
  • Health Checks: Built-in monitoring and recovery

📊 Data Quality

  • Dual Search: Award + Revenue flights in one run
  • CPP Calculation: Automatic cents-per-point value
  • Flight Matching: Merges Award + Revenue for same flights
  • Raw Data Saved: Full API responses preserved
  • Structured Output: Clean JSON format

🛠️ Configuration

Environment Variables

# Cookie directory (default: /app/cookies)
COOKIES_DIR=/path/to/cookies

# Output directory (default: /app/output)
OUTPUT_DIR=/path/to/output

# Log directory (default: /app/logs)
LOGS_DIR=/path/to/logs

# Browser path (auto-detected)
PLAYWRIGHT_BROWSERS_PATH=/path/to/browsers

CLI Arguments

Flight Search:
  --origin ORIGIN              Origin airport code
  --destination DESTINATION    Destination airport code
  --date DATE                  Departure date YYYY-MM-DD
  --origins ORIGIN [ORIGIN ...]        Multiple origins for bulk search
  --destinations DEST [DEST ...]       Multiple destinations for bulk search
  --dates DATE [DATE ...]              Multiple dates for bulk search
  --passengers N               Number of passengers (default: 1)
  --cabin CLASS                Cabin class (default: COACH)
  --search-type TYPE           Award, Revenue, or both (default: both)
  --max-concurrent N           Max concurrent combinations (default: 10)

Cookie Management:
  --cookies FILE               Cookie file path
  --no-headless                Show browser during cookie extraction
  --cookie-wait-time N         Wait time for API response (default: 15s)

Configuration:
  --output DIR                 Output directory (default: ./output)
  --rate-limit N               Requests per second (default: 1.0)
  --verbose                    Enable debug logging
  --log-file FILE              Log file path

🔍 How It Works

  1. Cookie Extraction (automatic on first run)

    • Launches Camoufox browser (real Firefox)
    • Navigates to AA.com and accepts cookies
    • Triggers a test flight search to warm up session
    • Captures cookies, headers, and referer
    • Validates API response before saving

  2. Bulk Scraping Mode (when using --origins, --destinations, or --dates)

    • Generates all origin × destination × date combinations
    • Creates semaphore to limit concurrent tasks
    • Launches tasks up to --max-concurrent limit
    • Each task independently searches Award + Revenue
    • Shared cookie pool across all tasks
    • Automatic error handling and retries per task

  3. Flight Search (per route/date combination)

    • Uses extracted cookies to make API requests
    • Searches Award flights (miles + taxes)
    • Searches Revenue flights (cash prices)
    • Matches flights by time/route
    • Calculates CPP (cents per point) value

  4. Cookie Reuse

    • Cookies cached for 30 minutes
    • Auto-refresh when expired
    • Retry logic on 403 errors
    • Exponential backoff on rate limits

  5. Data Merging

    • Matches Award + Revenue flights by departure/arrival times
    • Calculates true award value (CPP)
    • Filters by requested cabin class
    • Saves structured JSON output per combination

⚠️ Important Notes

Rate Limiting

  • Default: 1 request per second (safe)
  • AA.com has bot detection - don't go too fast
  • Circuit breaker opens after 3 consecutive failures
  • Automatic backoff on 429 responses
  • Bulk mode: Each concurrent task respects rate limit independently

Bulk Scraping Guidelines

  • Start conservative: --max-concurrent 5 and --rate-limit 1.0
  • Monitor logs: Watch for 403/429 errors
  • Increase gradually: If successful, try 10-15 concurrent
  • Cookie efficiency: All tasks share same cookies (good!)
  • RAM usage: Each task ~100-200MB, so 10 concurrent ≈ 1-2GB
  • One cookie extraction: Browser only launches once, not per task

Cookie Lifespan

  • Cookies expire after 30 minutes
  • Auto-refresh triggers at 20 minutes
  • Force refresh on 403 errors
  • Test flight validates cookies work

Akamai Challenges

  • Automatically detected and solved
  • May add 10-30s to first request
  • Transparent handling - no user action needed

Docker Volumes

Always mount these directories:

  • /app/cookies - Cookie persistence
  • /app/output - Results storage
  • /app/logs - Log files

Without volumes, data is lost when container stops!

🐛 Troubleshooting

"No cookies found" or "Cookie extraction failed"

Solution 1: Increase wait time

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --cookie-wait-time 30

Solution 2: Run with visible browser to see what's happening

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --no-headless

"403 Forbidden" errors

This means cookies are invalid or flagged. The scraper auto-refreshes cookies, but you can force it:

# Delete cached cookies
rm -rf cookies/*

# Run again - fresh cookies will be extracted
docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15

"Circuit breaker open"

Too many failures detected. Wait 5 minutes or adjust rate limit:

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --date 2025-12-15 \
  --rate-limit 0.5  # Slower = safer

Bulk scraping too many 429s (rate limit)

Reduce concurrency or rate limit:

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 5 \
  --rate-limit 0.8

Check logs

# View live logs
tail -f logs/aa_scraper.log

# Search for errors
grep ERROR logs/aa_scraper.log

# Check successful completions
grep "✅ Completed" logs/aa_scraper.log

📊 Example Use Cases

Trip Planning: Check Multiple Routes

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destinations JFK MIA ORD DFW ATL \
  --date 2025-12-15 \
  --search-type Award \
  --max-concurrent 5

Result: Compare award availability across 5 destinations in ~10 seconds!

Find Best Travel Dates

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destination JFK \
  --dates 2025-12-15 2025-12-16 2025-12-17 2025-12-18 2025-12-19 2025-12-20 2025-12-21 \
  --max-concurrent 7

Result: Check 7 days of availability in ~15 seconds!

Multi-City Award Search

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO SEA \
  --destinations JFK BOS PHL \
  --date 2025-12-15 \
  --search-type Award \
  --cabin BUSINESS \
  --max-concurrent 9

Result: 3×3 = 9 combinations, find best west-to-east coast business class award!

Monitor Availability Over Time (Cron)

# Add to crontab: Check daily at 8am
0 8 * * * docker run --rm \
  -v "$HOME/aa-scraper/cookies:/app/cookies" \
  -v "$HOME/aa-scraper/output:/app/output" \
  -v "$HOME/aa-scraper/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX \
  --destinations JFK MIA \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 4

Analyze Best CPP Across Routes and Dates

# Scrape comprehensive dataset
docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-20 2025-12-25 \
  --max-concurrent 5

# Find best CPP values
grep -r "cpp" output/*_combined.json | \
  jq -r '.flights[] | "$$.cpp) - $$.segments.departure_time) $$.segments[-1].arrival_time)"' | \
  sort -n -r | head -20

🏗️ Architecture

aa_scraper/
├── __init__.py              # Package initialization
├── __main__.py              # Entry point
├── cli.py                   # Command-line interface + bulk scraping
├── config.py                # Configuration constants
├── api_client.py            # HTTP client with retry logic
├── cookie_manager.py        # Cookie extraction & management
├── parser.py                # API response parser
├── storage.py               # Results persistence
├── rate_limiter.py          # Adaptive rate limiting
├── circuit_breaker.py       # Circuit breaker pattern
├── retry.py                 # Exponential backoff
├── exceptions.py            # Custom exceptions
├── logging_config.py        # Loguru setup
└── models.py                # Data models & enums

Key Components:

  • scrape_bulk_concurrent(): Orchestrates parallel route/date scraping
  • CookieManager: Browser automation with Camoufox, handles Akamai
  • AAFlightClient: HTTP/2 client with auto-recovery
  • AdaptiveRateLimiter: Token bucket algorithm with backoff
  • CircuitBreaker: Prevents cascading failures
  • FlightDataParser: Extracts flight data from API JSON

📈 Performance

Single Route Times:

  • Cookie extraction: 15-30 seconds (first run only)
  • Award search: 2-3 seconds (with cached cookies)
  • Revenue search: 2-3 seconds
  • Total (both): ~5-6 seconds after initial cookie setup

Bulk Scraping Performance:

  • 10 routes concurrently: ~6-8 seconds total (vs 50-60s sequential)
  • 24 combinations (2×4×3): ~12-15 seconds (vs 2-3 minutes sequential)
  • 50 combinations (5×5×2): ~25-30 seconds (vs 4-5 minutes sequential)
  • Speedup: 10-20x faster than sequential

Throughput:

  • Default rate limit: 1 req/s per task (safe for production)
  • Burst capacity: 2 requests per task
  • Handles 429 rate limits automatically
  • Recommended max concurrent: 10-15 tasks

Success Rate:

  • Cookie extraction: >95% (with retries)
  • API searches: >98% (with valid cookies)
  • Akamai challenge pass rate: 100% (automated)
  • Bulk scraping: >95% per combination (with error isolation)

🤝 Contributing

Contributions welcome! This is a production-ready tool, so please:

  1. Test thoroughly before submitting PRs
  2. Follow existing patterns (circuit breaker, retry logic, etc.)
  3. Update tests for new features
  4. Document changes in README

📄 License

MIT License - see LICENSE file

⚡ Quick Reference

Docker - Single Route

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origin LAX --destination JFK --date 2025-12-15

Docker - Bulk Scraping

docker run --rm \
  -v "$(pwd)/cookies:/app/cookies" \
  -v "$(pwd)/output:/app/output" \
  -v "$(pwd)/logs:/app/logs" \
  thermostatic/aa-scraper:latest \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 5

Local (uv) - Single Route

uv run -m aa_scraper \
  --origin LAX --destination JFK --date 2025-12-15

Local (uv) - Bulk Scraping

uv run -m aa_scraper \
  --origins LAX SFO \
  --destinations JFK MIA ORD \
  --dates 2025-12-15 2025-12-16 \
  --max-concurrent 5

Docker Hub 🐳 https://hub.docker.com/repository/docker/thermostatic/aa-scraper

Made with ❤️ for award travel enthusiasts

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages