Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
May 16, 2025 - Python
#
web-scraping
Here are 6,542 public repositories matching this topic...
The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
notifications monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection changedetection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
-
Updated
May 18, 2025 - Python
Python scraper based on AI
markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping
-
Updated
May 16, 2025 - Python
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
-
Updated
May 19, 2025 - TypeScript
🔥 Open Source No Code Web Data Extraction Platform • Turn Websites To APIs & Spreadsheets With No-Code Robots In Minutes 🔥
api scraper automation browser web-scraper self-hosted web-scraping data-extraction webscraping agents browser-automation no-code web-automation rpa robotic-process-automation web-agent playwright website-to-api no-code-web-scraper web-scraping-agent
-
Updated
May 19, 2025 - TypeScript
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
-
Updated
Mar 23, 2025 - Python
Python APIs for web automation, testing, and bypassing bot-detection.
python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
-
Updated
May 16, 2025 - Python
Lighter web automation with Python
-
Updated
Apr 28, 2025 - Python
List of libraries, tools and APIs for web scraping and data processing.
crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python
-
Updated
Dec 27, 2024 - Makefile
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
-
Updated
Oct 12, 2024 - Python
A Chrome DevTools Protocol driver for web automation and scraping.
testing go golang scraper automation web chrome-devtools headless devtools crawling web-scraping cdp chrome-headless rod chrome-devtools-protocol devtools-protocol gorod
-
Updated
Dec 7, 2024 - Go
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
May 19, 2025 - Python
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping
-
Updated
May 18, 2025 - Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm
-
Updated
Mar 17, 2025 - Python
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
http curl https http-client web-scraping fingerprinting ja3 tls-fingerprint curl-impersonate ja3-fingerprint http2-fingerprint akamai-fingerprint
-
Updated
May 18, 2025 - Python
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api php http client json framework curl xml proxy restful class http-client http-proxy api-client web-scraper requests web-scraping php-curl web-service php-curl-library
-
Updated
May 12, 2025 - PHP
Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.
web-crawler web-scraping data-collection batch-processing content-extraction search-api claude llm-tools firecrawl model-context-protocol mcp-server firecrawl-ai javascript-rendering
-
Updated
May 16, 2025 - JavaScript
Self-hosted webscraper.
python docker kubernetes opensource helm scraping webscraper web-scraper self-hosted web-scraping web-scrapers webscraping playwright
-
Updated
May 18, 2025 - TypeScript
The Python Code Tutorials
python python-tutorials machine-learning natural-language-processing computer-vision text-classification tutorials python3 web-scraping face-detection scapy network-analysis network-programming programming-tutorial ethical-hacking network-security socket-programming scapy-tutorials
-
Updated
Apr 14, 2025 - Jupyter Notebook
Improve this page
Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."