Async Python Web Crawler

A concurrent web crawler built for the Boot.dev Build a Web Scraper in Python guided project. It follows links within one domain, extracts useful page metadata, and writes a sorted JSON report.

Features

• Asynchronous crawling with aiohttp
• Configurable concurrency and page limits
• Same-domain link filtering and normalized URL deduplication
• Heading, first-paragraph, link, and image extraction with Beautiful Soup
• Deterministic JSON reporting

Run it

Install the project with uv, then provide a URL, maximum concurrency, and maximum page count:

uv sync
uv run main.py https://example.com 3 25

The crawler saves its results to report.json.

Tests

uv run -m unittest