A robust, production-grade sticker scraping pipeline for the StickerFight game. This tool uses Firecrawl to crawl and extract stickers (PNG, WebP, GIF) from publicly available sources, deduplicates them using SHA-256 fingerprinting, and outputs a structured CSV ready for bulk import into the game's MongoDB sticker collection.
- Headless Scraping: Uses Firecrawl to scrape client-side rendered HTML and lazy-loaded images.
- Deduplication: SHA-256 fingerprinting for image URLs to prevent duplicate indexing across runs.
- CSV Output: Append-safe
utf-8-sigCSV generation for Microsoft Excel compatibility. - Models & Validation: Pydantic models for strict sticker record schemas.
- Python 3.10+
- A Firecrawl API Key
-
Install dependencies:
pip install -r requirements.txt
-
Environment Variables: Copy
.env.exampleto.envand configure your API keys:FIRECRAWL_API_KEY=fc-your-api-key-here
Run the module via CLI and target a specific URL:
python -m sticker_scraper.main --url "https://target-sticker-site.com/pack" --output ./data/my_stickers.csv