A tool can scrape product in aliexpress: Title, Price, and URL Product.

Mining URLs from dark corners of Web Archives for bug hunting/fuzzing/further probing

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Fast directory scanning and scraping tool

A Python module to bypass Cloudflare's anti-bot page.

Example basic web crawler built on top of Pyppeteer

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

📦爬虫工具 【自动识别 验证码 12306、TX、Sina、Sogou 等】【免费短信接收】【一键获取代理IP】【正则匹配测试】【一键转码】【HASH】【IP查询】【网页调试】喜欢的话请 star 支持一下

溯光 (TrackRay) 3 beta⚡渗透测试框架（资产扫描|指纹识别|暴力破解|网页爬虫|端口扫描|漏洞扫描|代码审计|AWVS|NMAP|Metasploit|SQLMap）

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Scrapy, a fast high-level web crawling & scraping framework for Python.

🌈Python3网络爬虫实战：淘宝、京东、网易云、B站、12306、抖音、笔趣阁、漫画小说下载、音乐电影下载等

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

Elegant Scraper and Crawler Framework for Golang

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Using Socks4/5 or http proxies to make a multithreading Http-flood/Https-flood (cc) attack.