Skip to content

owgit/seo-crawler

Repository files navigation

SEO Migration Tracker

Enterprise tool for website platform migrations. Scan before, scan after, compare differences.

Use Case

Problem: When migrating to a new platform, SEO elements break silently (titles, meta, canonicals, URLs).

Solution:

  1. Scan old site → baseline
  2. Scan new site → post-migration
  3. Compare → see exactly what changed

What It Tracks

Field Severity
Missing URLs (404) Critical
New noindex Critical
Status code changes High
Title changes High
Canonical changes High
Hreflang changes High
Redirect changes High
Meta description Medium
H1 changes Medium
OG tags Medium
Image alt text Low-Medium

Quick Start (Docker)

# Production
docker-compose up --build

# Dev mode (hot-reload)
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build

# Raspberry Pi (ARM)
docker-compose -f docker-compose.rpi.yml up -d --build
Environment Frontend API
Mac/Local :3000 :8000
RPI :3002 :8001

Quick Start (Lokal dev)

# 1. Endast databaser i Docker
docker-compose up postgres redis -d

# 2. Backend
cd apps/api && python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && playwright install chromium
python db.py upgrade
uvicorn main:app --reload

# 3. Frontend (ny terminal)
cd apps/web && npm install && npm run dev

Database

cd apps/api

# Vanliga kommandon
python db.py upgrade          # Applicera nya migrationer
python db.py current          # Visa nuvarande version
python db.py history          # Visa alla migrationer

# Utveckling
python db.py new "message"    # Skapa ny migration
python db.py downgrade        # Backa en version
python db.py reset            # Radera allt, skapa om (DEV ONLY)

# Existerande databas → Alembic
python db.py stamp 002        # Markera DB vid revision utan att köra migration

Stack

  • Backend: FastAPI + PostgreSQL + Redis
  • Frontend: Next.js 14 + React Query
  • Crawler: aiohttp + Playwright (JS rendering)

API

POST /api/crawl/{project_id}/start    # Start crawl
GET  /api/compare/{a}/{b}             # Compare two crawls
GET  /api/compare/{a}/{b}/export      # CSV export
GET  /api/compare/{a}/{b}/summary     # Quick summary

👤 Author

Uygar Duzgun

Website GitHub Buy Me a Coffee

About

AI-native SEO crawler - alternative to Screaming Frog

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors