Releases: D4Vinci/Scrapling
Release list
Release v0.4.14
A quick maintenance release to fix installation with uv 🔧
🐛 Bug Fixes
- Fixed
uvrefusing to install v0.4.13 by default and silently falling back to an older version. The previous release required a prerelease version ofcurl_cffi, whichuvdoesn't allow unless explicitly enabled.
All dependencies now resolve to stable releases. (Fixes #407)
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.13
A new update bringing feed spiders and a smarter MCP server 🎉
Note
- Follow us on X for daily tips and tricks
- This will most likely be the last update before the major updates in v0.5
🚀 New Stuff and quality of life changes
- New feed spider templates.
XMLFeedSpideriterates over the nodes of any XML feed (RSS, Atom, product feeds, etc.), andCSVFeedSpideriterates over CSV rows as dictionaries. Both decompress gzipped feeds automatically. (Check the docs)from scrapling.spiders import XMLFeedSpider class RSSSpider(XMLFeedSpider): name = "rss" start_urls = ["https://example.com/feed.xml"] async def parse_node(self, response, node): yield {"title": node.findtext("title"), "link": node.findtext("link")} result = RSSSpider().start()
- Upgraded the MCP server to MCP SDK v2 and made it smarter. The server now ships instructions that teach your AI agent how to use the tools efficiently; every tool declares annotations so clients like Claude Code can auto-approve the read-only ones; tool descriptions are leaner to save tokens; and the server advertises its version and logo to MCP clients. (Check the docs)
- Added a
scrapling-mcpcommand that maps directly toscrapling mcp, so registering Scrapling with MCP clients and registries that expect a single command is now a one-liner. - Unpinned Playwright/Patchright and browser versions. The generated browser User-Agent now always matches the exact Chromium version your installed Playwright/Patchright drives, so Scrapling no longer pins their versions and you can upgrade them freely. Run
scrapling install --forceafter updating to refresh the browsers.
🐛 Bug Fixes
- Fixed importing Scrapling crashing with a
browserforgeValueError when the fingerprints data package lags behind the browser versions. (Fixes #394, #396, and #400) - Fixed the MCP bulk browser tools mis-sizing their page pools, which made
bulk_fetchfail on batches of more than 50 URLs andbulk_stealthy_fetchfetch all URLs through a single tab, by @Yigtwxx in #393.
Project
- New AI Contribution Policy: AI-assisted contributions are welcome but must be disclosed in the PR or issue; submissions that look like undisclosed AI output get labeled and closed.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.12
A release focused on making your spiders smarter about the websites they crawl
🚀 New Stuff and quality of life changes
-
Spiders can now tune their own speed with AutoThrottle. Instead of guessing a
download_delaythat's either too slow or gets you banned, the spider measures how fast each website answers and adjusts the delay of every domain on its own. When a website starts blocking or rate-limiting you, it doubles the delay (or waits exactly what theRetry-Afterheader asks for) until that stops, then speeds back up. Yourdownload_delayand any robots.txtCrawl-delayare still respected as the minimum. (Check the docs)class MySpider(Spider): name = "adaptive" start_urls = ["https://example.com"] autothrottle_enabled = True autothrottle_start_delay = 2.0 autothrottle_max_delay = 30.0 autothrottle_block_backoff = True
-
Export your results to CSV and XML, next to the JSON/JSONL exporters you already had. Items that don't all share the same keys are still exported without losing anything, and nested values are written as JSON. (Check the docs)
result = MySpider().start() result.items.to_csv("products.csv") result.items.to_xml("products.xml")
-
The MCP server can now require authentication, so you can safely expose it instead of keeping it on your own machine. Any request without the token is rejected, and you can also restrict which hostnames the server answers to. (Check the docs)
scrapling mcp --http --auth-token "$(openssl rand -hex 32)" -
Browsers now accept CDP URLs over HTTP, not just WebSocket ones. So next to the
wss://endpoints managed browser providers hand out, you can now point any browser fetcher or MCP session at a Chrome you started yourself with--remote-debugging-port=9222. -
Published Docker images are now tagged with their release version instead of only
latest, so you can pin the exact version you want, by @JanRK in #384.
🐛 Bug Fixes
-
Fixed cached responses losing all their cookies when the response came from a browser engine, which silently broke any session or auth logic relying on them while using the spiders' development mode, by @amitvijapur in #379. (Fixes #376)
-
Fixed
StealthyFetcherforcing theen-USlocale on every browser instead of following your system's, which made websites see a mismatch between your locale and your IP address and treat you as suspicious, like Google answering with 429s. (Fixes #381) -
Fixed a misleading error message in the storage system and removed a dead call left after inserts, by @fix2015 in #377.
Performance
get_all_text()is now O(nodes) instead of walking up the ancestors of every single text node, which makes it around 5-6x faster on deeply nested pages, by @yetval in #378.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.11
A solid update bringing the first platform spider template, a faster parser, and important fixes 🎉
🚀 New Stuff and quality of life changes
- Added
ShopifySpider, the first platform spider template! Extract every product from any Shopify-powered store through its JSON API without touching the website's HTML. Subclass it, set the store's domain, and you are done (Check the docs)from scrapling.spiders import ShopifySpider class MyStore(ShopifySpider): target_website = "example.com" result = MyStore().start()
- Added
--executable-pathto the CLI browser commands. Bothscrapling extract fetchandscrapling extract stealthy-fetchnow accept a custom Chromium-compatible browser executable, and fall back to theSCRAPLING_EXECUTABLE_PATHenvironment variable when the option isn't passed, bringing full parity with the MCP server (Solves #371)scrapling extract fetch "https://example.com" page.html --executable-path "/path/to/chromium"
🤖 Quality of life changes
- Made
find_by_textandfind_by_regexup to ~2x faster whenfirst_matchis enabled (the default) by wrapping elements lazily so the search stops at the first match, by @yetval in #370 - Updated the benchmarks with the new numbers against the latest versions of all libraries.
- Updated contribution rules
🐛 Bug Fixes
- Fixed the MCP server's fetch tools crashing on pages containing control characters with the error
All strings must be XML compatible, by @yetval in #368 (Fixes #366)
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors:
Contributors
Assets 2
Release v0.4.10
A new update with a brand-new Scrapy integration and a batch of community fixes 🎉
🚀 New Stuff and quality of life changes
-
Added a Scrapy integration so you can use Scrapling's parsing API inside your existing Scrapy projects without rewriting them. Put the
scrapling_responsedecorator on any spider callback, and the response it receives becomes a ScraplingResponsewhile Scrapy keeps handling the crawling (Check the docs):import scrapy from scrapling.integrations.scrapy import scrapling_response class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = ["https://quotes.toscrape.com"] @scrapling_response def parse(self, response): # `response` is now a Scrapling Response first_quote = response.find_by_text("The world as we have created it", partial=True) for quote in [first_quote, *first_quote.find_similar()]: yield {"text": quote.get_all_text(strip=True)}
-
The MCP server can now use a custom Chromium-compatible browser for all browser-based tools. Set it once with
scrapling mcp --executable-path "/path/to/chromium"or theSCRAPLING_EXECUTABLE_PATHenvironment variable, or per request with theexecutable_pathargument, by @samrusani in #360 (Solves #347) -
Updated all browsers and fingerprints. Run
scrapling install --forceafter updating to refresh them.
🐛 Bug Fixes
- Fixed garbled text (mojibake) from browser fetchers on non-UTF-8 websites by @yehudalevy-collab in #365 (Fixes #364).
- Fixed
LinkExtractornot filtering compound file extensions like.tar.gzby @renbkna in #359 (Fixes #349). - Fixed paused crawls losing their in-flight requests from checkpoints, so resuming no longer skips them by @yetval in #358.
- Fixed spiders calculating wrong crawl delays from robots.txt
Request-ratedirectives through the Protego upgrade, with tests aligned by @Disaster-Terminator in #355.
Docs
- Clarified how
init_scriptinteracts with Patchright's isolated execution context in stealth mode by @mturac in #353 (Solves #350). - Added the skills.sh install method for the agent skill by @ob-aion in #363.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.9
A maintenance update packed with community-reported fixes 🛠️
🚀 New Stuff and quality of life changes
- Updated all browsers and fingerprints. Run
scrapling install --forceafter updating to refresh them. - Added a
--versionflag to the CLI by @ETM-Code in #303 (Solves #299)
🐛 Bug Fixes
- Fixed the session-level
proxyargument being silently ignored in HTTP sessions, which could leak your real IP (Solves #295). Note that mixing a session-levelproxywith a per-requestproxiesargument (or vice versa) now raises an error instead of one being silently dropped. - Fixed browser navigations failing when combining
init_scriptwithuser_data_dir(Solves #294). - Fixed encoding detection when websites quote the charset value in the
Content-Typeheader by @Bortlesboat in #323. - Fixed an
IndexErrorin adaptive element relocation whenauto_saveis enabled by @Mubashirrrr in #340. - Fixed spiders' checkpoint and cache saving crashing on Windows by @MrStarkEG in #344.
- Fixed incorrect similarity scoring in
find_similarfor elements with mismatched attribute counts (Solves #322).
Docs
- Clarified that the default installation includes the parser engine only, and the fetchers/spiders need the extras (Solves #343).
- Fixed the Docker image name in the remaining examples by @evanclan in #315.
- Fixed a broken link in the contribution guide by @Bortlesboat in #320.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.8
A big spider update that takes the crawling framework to the next level 🕷️
🚀 New Stuff and quality of life changes
-
Added a
LinkExtractorprimitive inscrapling.spiders.LinkExtractorto pull URLs out of aResponse. There are a lot of controls (Check the docs)from scrapling.spiders import LinkExtractor extractor = LinkExtractor(allow=r"/posts/", deny_domains=["ads.example.com"])
-
Added
CrawlSpiderandCrawlRulegeneric spider templates so you no longer have to hand-write the same "follow links matching this pattern" boilerplate. Overriderules()to return a list ofCrawlRuleobjects, each pairing aLinkExtractor. (Check the docs)from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor class QuotesSpider(CrawlSpider): name = "blog" start_urls = ["https://quotes.toscrape.com/"] def rules(self): return [ CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author), CrawlRule(LinkExtractor(allow=r"/page/\d+/")), # pagination, no callback ] async def parse_author(self, response): yield { "name": response.css(".author-title::text").get(), "birthday": response.css(".author-born-date::text").get(), "url": response.url, }
-
Added a
SitemapSpidertemplate that seeds a crawl directly from a sitemap, orrobots.txtURLs. Handles gzip-compressed sitemaps, and a lot of controls and options. URLs are dispatched via the crawl rules as shown above for CrawlSpider. (Check the docs)from scrapling.spiders import SitemapSpider, CrawlRule, LinkExtractor class NewsSitemap(SitemapSpider): name = "news" sitemap_urls = ["https://example.com/robots.txt"] def rules(self): return [ CrawlRule(LinkExtractor(allow=r"/articles/"), callback=self.parse_article), ] async def parse_article(self, response): yield {"url": response.url, "title": response.css("h1::text").get()}
-
Adaptive relocation now defaults to a 40% similarity threshold instead of
0across all methods. This will make the adaptive feature work better. When nothing crosses the threshold, a warning now tells you the top score it did see, so you can lowerpercentagedeliberately if needed. -
Updated all browsers and fingerprints. Run a new
scrapling install --forceafter updating to refresh the browsers and fingerprints.
🐛 Bug Fixes
- Fixed
Fetcher.configure(...)not applying to per-request calls. Same fix applied toAsyncFetcher. - Fixed incorrect request fingerprinting that caused duplicate requests in spiders by @yetval in #255.
- Fixed the Adaptive scraping engine staying silent on weak matches. Combined with the threshold change above, you now get a warning instead of a misleading "best guess" element when relocation fails.
Docs
- Refreshed older code examples across the documentation to match the current version.
- Improved the code copy-paste experience on the docs site and trimmed the agent skill so it uses fewer tokens per invocation.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.7
A focused update bringing eyes to your AI agents 📸
🚀 New Stuff and quality of life changes
- Added a
screenshotMCP tool that captures a page and returns it as a real MCPImageContentblock so the model can actually see it. The tool requires an open browser session, so you callopen_sessionfirst (eitherdynamicorstealthy) and pass thesession_idhere. Supports PNG and JPEG, full-page captures, JPEG quality, and the usual readiness controls (wait,wait_selector,network_idle,timeout). (implements #244) - Added a custom
session_idparameter toopen_sessionso you can name sessions meaningfully ("search","checkout") instead of the random 12-character hex default. By @hauntedhost in #243
🐛 Bug Fixes
- Fixed
FetcherSessionstate corruption and a lazy session close crash. By @yetval in #245 - Fixed
TypeError: Session.request() got an unexpected keyword argument 'block_ads'when using the CLI's--ai-targetedflag with HTTP commands. By @voidborne-d in #249 (Fixes #247)
Translations
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.6
A focused update on browser stealth, privacy, and developer experience 🔒
🚀 New Stuff and quality of life changes
- Added built-in ad blocking for browser fetchers. Pass
block_ads=Trueto block requests to ~3,500 known ad and tracker domains at the route interception level -- no DNS, no TCP, instant abort. Can be combined withblocked_domainsfor custom lists. The MCP server and CLI--ai-targetedmode enable this automatically to save tokens and speed up page loads.page = StealthyFetcher.fetch('https://example.com', block_ads=True)
- Added DNS-over-HTTPS support to prevent DNS leaks when using proxies. Pass
dns_over_https=Trueto route DNS queries through Cloudflare's DoH, so your real location isn't exposed through DNS resolution even when your HTTP traffic goes through a proxy.page = StealthyFetcher.fetch('https://example.com', proxy='http://proxy:8080', dns_over_https=True)
- Added
page_setupcallback for browser fetchers. A function that runs beforepage.goto(), letting you register event listeners, routes, or scripts that must be set up before the page navigates. Pairs withpage_action(which runs after navigation). (Solves #237)def capture_websockets(page): page.on("websocket", lambda ws: print(f"WS: {ws.url}")) page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)
- Added
--block-adsand--dns-over-httpsCLI options to bothfetchandstealthy-fetchcommands.
🐛 Bug Fixes
- Fixed
Secondstype alias rejecting float values. Passingwait=1.5ortimeout=500.0to browser fetchers would fail with a type error because the type alias incorrectly treatedfloatas metadata instead of a type. by @kuishou68 in #240 - Fixed duplicate ID segments in full-path selector generation. Elements with
idattributes had their selector appended twice when generating full CSS/XPath paths, producing selectors likebody > #main > #main > #target > #target. Also fixed full-path XPath emitting bare[@id='x']predicates (invalid XPath) instead of*[@id='x']. by @sjhddh in #241 - Fixed missing shell signature parameters. The interactive shell was missing
blocked_domains,block_ads,retries,retry_delay,capture_xhr,executable_path, anddns_over_httpsfrom its function signatures.
🙏 Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Contributors
Assets 2
Release v0.4.5
A focused update with one big quality-of-life feature for spider developers and a couple of important fixes 🎉
🚀 New Stuff and quality of life changes
-
Spider Development Mode: Iterating on a spider's
parse()logic used to mean re-hitting the target servers on every run, which is slow, noisy, and a great way to get rate-limited while you're still figuring out your selectors. The new development mode caches every response to disk on the first run and replays them from disk on every subsequent run, so you can tweak your callbacks and re-run as many times as you want without making a single network request. Enable it with one class attribute:class MySpider(Spider): name = "my_spider" start_urls = ["https://example.com"] development_mode = True async def parse(self, response): yield {"title": response.css("title::text").get("")}
The cache lives in
.scrapling_cache/{spider.name}/by default and can be redirected anywhere withdevelopment_cache_dir. Two new stat counters,cache_hitsandcache_misses, let you see how the cache performed. Cache replay bypassesdownload_delay, rate limiting, and the blocked-request retry path so iteration is as fast as the disk allows. Don't ship a spider withdevelopment_mode = True-- it's a development tool, not a production cache. See the docs for the full story. -
Safer redirects by default:
follow_redirectsnow defaults to"safe"across all HTTP fetchers, the MCP server, and the shell. Redirects are still followed, but ones targeting internal/private IPs (loopback, private networks, link-local) are rejected. This protects you from SSRF when scraping user-supplied URLs. Passfollow_redirects="all"to get the old behavior, orFalseto disable redirects entirely.
🐛 Bug Fixes
- Force-stop no longer loses your checkpoint: Pressing Ctrl+C twice (force-stop) on a spider with
crawldirenabled used to race against the checkpoint write -- the cancel scope would tear down the task before the pickle finished, leavingpaused=Falseand triggering the cleanup path that deletes the previous checkpoint. The result was that force-stopping a long crawl could lose all the progress you were trying to save. The engine now writes the checkpoint before callingcancel_scope.cancel(), so a force-stop always preserves the latest pending state. By @voidborne-d in #230.
🙏 Special thanks to the community for all the continuous testing and feedback