CrawleePython
title: Crawlee for Python radar_quadrant: Languages & Frameworks radar_ring: Assess radar_position: inner
Crawlee for Python is an open-source web scraping and crawling framework developed by Apify. The project is available at github.com/apify/crawlee-python. It provides a unified API for both HTTP-based crawling (BeautifulSoup, HTTPX) and browser-based crawling (Playwright), with built-in handling for retries, proxy rotation, rate limiting, session management, and pluggable storage adapters (local filesystem, cloud).
The framework's core abstraction is the crawler class: BeautifulSoupCrawler for static HTML pages, PlaywrightCrawler for JavaScript-rendered pages. Both share the same request queue, storage, and configuration model — switching from one to the other requires changing the crawler class, not rewriting the scraping logic. Storage adapters write scraped data to a dataset (JSON records) or a key-value store; local adapters are the default, with Apify cloud adapters available for scale-out.
Placed in Languages & Frameworks / Assess / inner.
Scrapy (Languages/Assess/inner) covers static HTML scraping well; its Playwright integration is a third-party plugin and a secondary concern. Crawlee treats browser automation as first-class: the PlaywrightCrawler is a primary path, not an add-on. For the large proportion of modern sites that render content via JavaScript after page load, this is a meaningful capability difference.
The operational primitives — request queue with deduplication, automatic retries, proxy rotation, rate limit handling — address the failure modes that make production scrapers brittle. These are implemented in the framework rather than re-built per project.
The positioning relative to Firecrawl (Tools/Assess/inner): Firecrawl is a managed API for zero-setup site-scale crawling; Crawlee is a library for scrapers you own, iterate on, and control fully. They serve different operational models — Crawlee for custom pipelines with specific extraction logic, Firecrawl for rapid ingestion without custom code.
Inner position reflects direct applicability to any Python scraping project requiring JS-rendered pages, composability with Data Contracts for Scraping Pipelines (schema validation at output), Git-backed SQLite Data Pipeline (storage destination), and Scrapy (migration path for existing projects).
Trial gate: a Crawlee PlaywrightCrawler used to extract structured data from at least one JS-rendered site, with output validated by a Pydantic schema.
