Image Scraper

A multi-threaded, crash-resilient scraper that bulk-downloads high-resolution images from large CSV/XLSX datasets. Built for runs of 10,000–200,000+ rows with full resume, auto-retry, and per-row checkpointing.

Comes with a Next.js web UI for uploading datasets, configuring jobs, and watching progress in real time.

Features

• Parallel workers — N independent Chromium browsers running simultaneously
• Resumable — crash at row 47,832, restart exactly from row 47,832
• Auto-retry pipeline — strips failed rows and re-runs until zero failures or stall
• Smart image selection — scores candidates from DOM, JSON-LD, meta tags, network interception, and dataset URLs
• CDN upgrades — rewrites thumbnail URLs to full resolution automatically
• Anti-bot fingerprint — Chrome 122 UA, en-IN locale, JS patches, optional playwright-stealth
• Column auto-detection — works on unknown CSVs without any config

Installation

pip install pandas playwright openpyxl playwright-stealth
playwright install chromium

playwright-stealth is optional but recommended.

Usage

Full pipeline (recommended)

Runs the scraper, then automatically retries failed rows up to N rounds.

python run_image_pipeline.py data.csv --workers 4 --delay 1.0 --max-retry-rounds 2

Raw scraper

python scraper.py data.csv --workers 4 --delay 1.0

Resume after a crash

python scraper.py data.csv --workers 4 --resume

Test on a small slice

python scraper.py data.csv --rows 50

Custom column names

python scraper.py custom.csv --url-col "Page URL" --name-col "Title" --image-col "Thumbnail"

Diagnose a failing URL

Shows the ranked image candidates the scraper would evaluate — use when a site stops yielding images.

python diagnose.py "https://example.com/item-name"

Manual retry

python retry_failed.py data.csv
python scraper.py data.csv --workers 4 --resume

Output

Files land in a folder named after the input file stem (e.g. data.csv → data_images_output/).

File	Description
<name>_log.csv	One row per item: id, name, URLs, saved path, status, timestamp, worker
_checkpoint.txt	Completed row IDs — drives --resume
_invalid.xlsx	Failed rows exported by retry_failed.py for manual review

Options

Flag	Default	Description
--workers	4	Parallel browser workers
--delay	1.0	Per-worker pause between requests (seconds)
--max-retry-rounds	2	Auto-retry rounds in the pipeline
--rows	—	Limit rows (for testing)
--resume	off	Skip rows already in checkpoint
--output-dir	file stem	Override output folder

Using a different dataset

Option 1 — CLI flags (no code changes)

Pass column names directly:

python run_image_pipeline.py my_data.csv \
  --url-col "Page URL" \
  --name-col "Product Name" \
  --id-col "SKU" \
  --image-col "Thumbnail URL" \
  --workers 4

--image-col is optional. If your dataset has a direct image URL column, pass it — the scraper tries it first before loading the page. Omit it for page-crawl only.

Option 2 — Add an entry to DATASET_CONFIG

For datasets you run repeatedly, add a permanent mapping in scraper.py at the top (DATASET_CONFIG). The key is the exact filename:

"my_data.csv": {
    "url_col":   "Page URL",       # column containing the product/item page URL
    "name_col":  "Product Name",   # used for the saved filename
    "id_col":    "SKU",            # unique row identifier; used by --resume (set None to auto-generate)
    "image_col": "Thumbnail URL",  # direct image URL column; set None for page-crawl only
},

Option 3 — Auto-detection

If you pass neither CLI flags nor a DATASET_CONFIG entry, the scraper will try to detect URL, name, and image columns automatically. This works for simple CSVs but is less reliable for unusual column names.

Site-specific tuning

If images from a new site come out as thumbnails or the wrong resolution:

• upgrade_to_hires in image_discovery.py — add a URL rewrite rule to swap CDN thumbnail parameters for full-resolution ones.
• normalise_page_url in image_discovery.py — add a rewrite if the site redirects old-format URLs before the page finishes loading.

Run diagnose.py against a sample URL first to see what image candidates the scraper finds:

python diagnose.py "https://yoursite.com/product-page"

Web UI

A minimal Next.js interface for running jobs without touching the terminal.

Setup

cd frontend
npm install
npm run dev       # starts on localhost:3001

What it does

• Upload — drag-and-drop or browse for a .csv, .xlsx, or .xls file
• Configure — workers (1–8), per-request delay, optional row cap, resume mode
• Live progress — table streams item-level results via Server-Sent Events as the scraper runs
• Thumbnails — loads previews for all successful items in parallel using Promise.allSettled once the job finishes; individual failures don't block others
• Output path — shown on completion (<stem>_images_output/)

Stack

Next.js 14 (App Router) · React 18 · TypeScript · Tailwind CSS · SSE · Playwright subprocess

The API routes in frontend/app/api/ spawn the Python scraper as a detached child process and tail the log CSV to push updates to the browser. No separate API server needed.

Platform notes

• macOS — SSL certificate verification is disabled globally; some target sites have broken cert chains.
• Windows — WindowsSelectorEventLoopPolicy is set at startup due to a Playwright + ProactorEventLoop conflict.