Effortless Web Scraping for the Modern Web

Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.

Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of TypeScript. One library, zero compromises.

Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.

import { StealthyFetcher } from 'scrapling/fetchers';
StealthyFetcher.adaptive = true;
const p = await StealthyFetcher.fetch('https://example.com', { headless: true, networkIdle: true }); // Fetch website under the radar!
let products = p.css('.product', { autoSave: true });                                  // Scrape data that survives website design changes!
products = p.css('.product', { adaptive: true });                                      // Later, if the website structure changes, pass `adaptive=true` to find them!

Or scale up to full crawls

import { Spider, Response } from 'scrapling/spiders';

class MySpider extends Spider {
  name = "demo";
  startUrls = ["https://example.com/"];

  async *parse(response: Response) {
      for (const item of response.css('.product')) {
          yield { title: item.css('h2::text').get() };
      }
  }
}

const spider = new MySpider();
spider.start();

Platinum Sponsors

🚀 Subscribe \| 💬 Discord	Scrapling's fetchers handle Cloudflare Turnstile, but for enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for: Akamai • DataDome • Kasada • Incapsula No browser automation. Simple API calls returning the exact cookies and headers these systems require.
	Hey, we built BirdProxies because proxies shouldn't be complicated or overpriced. Fast residential and ISP proxies in 195+ locations, fair pricing, and real support. Try our FlappyBird game on the landing page for free data! 🚀 Try Now \| 💬 Discord

_{Do you want to show your ad here? Click here}

Sponsors

_{Do you want to show your ad here? Click here and choose the tier that suites you!}

Key Features

Spiders — A Full Crawling Framework

🕷️ Robust Spider API: Define spiders with start_urls, async parse callbacks, and Request/Response objects.
⚡ Concurrent Crawling: Configurable concurrency limits, per-domain throttling, and download delays.
🔄 Multi-Session Support: Unified interface for HTTP requests, and stealthy headless browsers in a single spider — route requests to different sessions by ID.
💾 Pause & Resume: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
📡 Streaming Mode: Stream scraped items as they arrive via async for item in spider.stream() with real-time stats — ideal for UI, pipelines, and long-running crawls.
🛡️ Blocked Request Detection: Automatic detection and retry of blocked requests with customizable logic.
📦 Built-in Export: Export results through hooks and your own pipeline or the built-in JSON/JSONL with result.items.toJson() / result.items.toJsonl() respectively.

Advanced Websites Fetching with Session Support

HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
Dynamic Loading: Fetch dynamic websites with full browser automation through the DynamicFetcher class supporting Playwright's Chromium and Google's Chrome.
Anti-bot Bypass: Advanced stealth capabilities with StealthyFetcher and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
Session Management: Persistent session support with FetcherSession, StealthySession, and DynamicSession classes for cookie and state management across requests.
Proxy Rotation: Built-in ProxyRotator with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
Domain Blocking: Block requests to specific domains (and their subdomains) in browser-based fetchers.
Async Support: Complete async support across all fetchers and dedicated async session classes.

Adaptive Scraping & AI Integration

🔄 Smart Element Tracking: Relocate elements after website changes using intelligent similarity algorithms.
🎯 Smart Flexible Selection: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
🔍 Find Similar Elements: Automatically locate elements similar to found elements.
🤖 MCP Server to be used with AI: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. (demo video)

High-Performance & battle-tested Architecture

🚀 Lightning Fast: Optimized performance outperforming most TypeScript scraping libraries.
🔋 Memory Efficient: Optimized data structures and lazy loading for a minimal memory footprint.
⚡ Fast JSON Serialization: 10x faster than the standard library.
🏗️ Battle tested: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.

Developer/Web Scraper Friendly Experience

🎯 Interactive Web Scraping Shell: Optional built-in REPL with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
🚀 Use it directly from the Terminal: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
🛠️ Rich Navigation API: Advanced DOM traversal with parent, sibling, and child navigation methods.
🧬 Enhanced Text Processing: Built-in regex, cleaning methods, and optimized string operations.
📝 Auto Selector Generation: Generate robust CSS/XPath selectors for any element.
🔌 Familiar API: Similar to BeautifulSoup with the same pseudo-elements used in Parsel.
📘 Complete Type Coverage: Full type hints for excellent IDE support and code completion. The entire codebase is automatically written in TypeScript.
🔋 Ready Docker image: With each release, a Docker image containing all browsers is automatically built and pushed.

Getting Started

Let's give you a quick glimpse of what Scrapling can do without deep diving.

Basic Usage

HTTP requests with session support

import { Fetcher, FetcherSession } from 'scrapling/fetchers';

const session = new FetcherSession({ impersonate: 'chrome' }); // Use latest version of Chrome's TLS fingerprint
const page = await session.get('https://quotes.toscrape.com/', { stealthyHeaders: true });
const quotes = page.css('.quote .text::text').getAll();
await session.close();

// Or use one-off requests
const page2 = await Fetcher.get('https://quotes.toscrape.com/');
const quotes2 = page2.css('.quote .text::text').getAll();

Advanced stealth mode

import { StealthyFetcher, StealthySession } from 'scrapling/fetchers';

const session = new StealthySession({ headless: true, solveCloudflare: true }); // Keep the browser open until you finish
const page = await session.fetch('https://nopecha.com/demo/cloudflare', { googleSearch: false });
const data = page.css('#padded_content a').getAll();
await session.close();

// Or use one-off request style, it opens the browser for this request, then closes it after finishing
const page2 = await StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare');
const data2 = page2.css('#padded_content a').getAll();

Full browser automation

import { DynamicFetcher, DynamicSession } from 'scrapling/fetchers';

const session = new DynamicSession({ headless: true, disableResources: false, networkIdle: true }); // Keep the browser open until you finish
const page = await session.fetch('https://quotes.toscrape.com/', { loadDom: false });
const data = page.xpath('//span[@class="text"]/text()').getAll(); // XPath selector if you prefer it
await session.close();

// Or use one-off request style, it opens the browser for this request, then closes it after finishing
const page2 = await DynamicFetcher.fetch('https://quotes.toscrape.com/');
const data2 = page2.css('.quote .text::text').getAll();

Spiders

Build full crawlers with concurrent requests, multiple session types, and pause/resume:

import { Spider, Response } from 'scrapling/spiders';

class QuotesSpider extends Spider {
    name = "quotes";
    startUrls = ["https://quotes.toscrape.com/"];
    concurrentRequests = 10;
    
    async *parse(response: Response) {
        for (const quote of response.css('.quote')) {
            yield {
                text: quote.css('.text::text').get(),
                author: quote.css('.author::text').get(),
            };
        }
            
        const nextPage = response.css('.next a');
        if (nextPage) {
            yield response.follow(nextPage[0].attrib['href']);
        }
    }
}

const spider = new QuotesSpider();
const result = await spider.start();
console.log(`Scraped ${result.items.length} quotes`);
result.items.toJson("quotes.json");

Use multiple session types in a single spider:

import { Spider, Request, Response } from 'scrapling/spiders';
import { FetcherSession, AsyncStealthySession } from 'scrapling/fetchers';

class MultiSessionSpider extends Spider {
    name = "multi";
    startUrls = ["https://example.com/"];
    
    configureSessions(manager: any) {
        manager.add("fast", new FetcherSession({ impersonate: "chrome" }));
        manager.add("stealth", new AsyncStealthySession({ headless: true }), { lazy: true });
    }
    
    async *parse(response: Response) {
        for (const link of response.css('a::attr(href)').getAll()) {
            // Route protected pages through the stealth session
            if (link.includes("protected")) {
                yield new Request(link, { sid: "stealth" });
            } else {
                yield new Request(link, { sid: "fast", callback: this.parse }); // explicit callback
            }
        }
    }
}

Pause and resume long crawls with checkpoints by running the spider like this:

const spider = new QuotesSpider({ crawlDir: "./crawl_data" });
spider.start();

Press Ctrl+C to pause gracefully — progress is saved automatically. Later, when you start the spider again, pass the same crawldir, and it will resume from where it stopped.

Advanced Parsing & Navigation

import { Fetcher } from 'scrapling/fetchers';

// Rich element selection and navigation
const page = await Fetcher.get('https://quotes.toscrape.com/');

// Get quotes with multiple selection methods
let quotes = page.css('.quote'); // CSS selector
quotes = page.xpath('//div[@class="quote"]'); // XPath
quotes = page.findAll('div', { 'class': 'quote' }); // BeautifulSoup-style
// Find element by text content
quotes = page.findByText('quote', { tag: 'div' });

// Advanced navigation
let quoteText = page.css('.quote')[0].css('.text::text').get();
quoteText = page.css('.quote').css('.text::text').getAll(); // Chained selectors
const firstQuote = page.css('.quote')[0];
const author = firstQuote.nextSibling.css('.author::text');
const parentContainer = firstQuote.parent;

// Element relationships and similarity
const similarElements = firstQuote.findSimilar();
const belowElements = firstQuote.belowElements();

You can use the parser right away if you don't want to fetch websites like below:

import { Selector } from 'scrapling/parser';

const page = new Selector("<html>...</html>");

And it works precisely the same way!

Async Session Management Examples

import { FetcherSession, AsyncStealthySession } from 'scrapling/fetchers';

const session = new FetcherSession({ http3: true }); // `FetcherSession` is context-aware and can work in both sync/async patterns
const page1 = await session.get('https://quotes.toscrape.com/');
const page2 = await session.get('https://quotes.toscrape.com/', { impersonate: 'firefox135' });
await session.close();

// Async session usage
const asyncSession = new AsyncStealthySession({ maxPages: 2 });
const tasks = [];
const urls = ['https://example.com/page1', 'https://example.com/page2'];

for (const url of urls) {
    tasks.push(asyncSession.fetch(url));
}

console.log(asyncSession.getPoolStats()); // Optional - The status of the browser tabs pool (busy/free/error)
const results = await Promise.all(tasks);
console.log(asyncSession.getPoolStats());
await asyncSession.close();

CLI & Interactive Shell

Scrapling includes a powerful command-line interface:

Launch the interactive Web Scraping shell

scrapling shell

Extract pages to a file directly without programming (Extracts the content inside the body tag by default). If the output file ends with .txt, then the text content of the target will be extracted. If it ends in .md, it will be a Markdown representation of the HTML content; if it ends in .html, it will be the HTML content itself.

scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome'  # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare

Note

There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation here

Performance Benchmarks

Scrapling isn't just powerful—it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.

Text Extraction Speed Test (5000 nested elements)

#	Library	Time (ms)	vs Scrapling
1	Scrapling	2.02	1.0x
3	Raw Lxml	2.54	1.257
5	Selectolax	82.63	~41x
6	MechanicalSoup	1549.71	~767.1x
7	BS4 with Lxml	1584.31	~784.3x
8	BS4 with html5lib	3391.91	~1679.1x

Element Similarity & Text Search Performance

Scrapling's adaptive element finding capabilities significantly outperform alternatives:

Library	Time (ms)	vs Scrapling
Scrapling	2.39	1.0x
AutoScraper	12.45	5.209x

All benchmarks represent averages of 100+ runs. See benchmarks.ts for methodology.

Installation

Scrapling requires TypeScript 3.10 or higher:

bun install scrapling

This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies.

Optional Dependencies

If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
```
bun install scrapling

scrapling install           # normal install
scrapling install  --force  # force reinstall
```
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.

Or you can install them from the code instead of running a command like this:
```
import { install } from "scrapling/cli";

install([], false);          // normal install
install(["--force"], false); // force reinstall
```
Extra features:
- Install the MCP server feature:
```
bun install scrapling
```
- Install shell features (Web Scraping shell and the extract command):
```
bun install scrapling
```
- Install everything:
```
bun install scrapling
```
Remember that you need to install the browser dependencies with scrapling install after any of these extras (if you didn't already)

Docker

You can also install a Docker image with all extras and browsers with the following command from DockerHub:

docker pull pyd4vinci/scrapling

Or download it from the GitHub registry:

docker pull ghcr.io/d4vinci/scrapling:latest

This image is automatically built and pushed using GitHub Actions and the repository's main branch.

Contributing

We welcome contributions! Please read our contributing guidelines before getting started.

Disclaimer

Caution

This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.

🎓 Citations

If you have used our library for research purposes please quote us with the following reference:

  @misc{scrapling,
    author = {Karim Shoair},
    title = {Scrapling},
    year = {2024},
    url = {https://github.com/D4Vinci/Scrapling},
    note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
  }

License

This work is licensed under the BSD-3-Clause License.

Acknowledgments

This project includes code adapted from:

Parsel (BSD License)—Used for translator submodule

Designed & crafted with ❤️ by Karim Shoair.

Name		Name	Last commit message	Last commit date
Latest commit History 1,187 Commits
src		src
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
ROADMAP.md		ROADMAP.md
bun.lock		bun.lock
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Navigation Menu

Repository files navigation

Effortless Web Scraping for the Modern Web

Platinum Sponsors

Sponsors

Key Features

Spiders — A Full Crawling Framework

Advanced Websites Fetching with Session Support

Adaptive Scraping & AI Integration

High-Performance & battle-tested Architecture

Developer/Web Scraper Friendly Experience

Getting Started

Basic Usage

Spiders

Advanced Parsing & Navigation

Async Session Management Examples

CLI & Interactive Shell

Performance Benchmarks

Text Extraction Speed Test (5000 nested elements)

Element Similarity & Text Search Performance

Installation

Optional Dependencies

Docker

Contributing

Disclaimer

🎓 Citations

License

Acknowledgments

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Effortless Web Scraping for the Modern Web

Platinum Sponsors

Sponsors

Key Features

Spiders — A Full Crawling Framework

Advanced Websites Fetching with Session Support

Adaptive Scraping & AI Integration

High-Performance & battle-tested Architecture

Developer/Web Scraper Friendly Experience

Getting Started

Basic Usage

Spiders

Advanced Parsing & Navigation

Async Session Management Examples

CLI & Interactive Shell

Performance Benchmarks

Text Extraction Speed Test (5000 nested elements)

Element Similarity & Text Search Performance

Installation

Optional Dependencies

Docker

Contributing

Disclaimer

🎓 Citations

License

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages