#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 6,036 public repositories matching this topic...

scrapy

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling hacktoberfest

Updated Jan 27, 2023
Python

iawia002 / lux

👾 Fast and simple video download library and CLI tool written in Go

go golang crawler scraper downloader youtube video download tumblr bilibili qq youku iqiyi

Updated Jan 13, 2023
Go

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

Updated Jan 16, 2023
Go

jhao104 / proxy_pool

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Jan 2, 2023
Python

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.

Updated Nov 20, 2022
Python

codelucas / newspaper

News, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

Updated Nov 15, 2022
Python

shengqiangzhang / examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report wereader

Updated Apr 27, 2022
Python

code4craft / webmagic

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Dec 20, 2022
Java

crawlab

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management

Updated Jan 15, 2023
Go

s0md3v / Photon

Sponsor

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Dec 26, 2022
Python

guyueyingmu / avbook

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Jan 4, 2023
PHP

injetlee / Python

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

python crawler excel wechat

Updated Jan 22, 2023
Python

ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Nov 20, 2022
Java

crawlee

apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

npm crawler scraper automation web-crawler headless scraping crawling javascript-library web-scraping web-crawling headless-chrome rpa apify puppeteer

Updated Jan 27, 2023
TypeScript

bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Jan 23, 2023
JavaScript

lorien / awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

Updated Dec 31, 2022
Makefile

chyroc / WechatSogou

Sponsor

基于搜狗微信搜索的微信公众号爬虫接口

python crawler pypi scrapy wechat sogou

Updated Nov 22, 2022
Python

projectdiscovery / katana

A next-generation crawling and spidering framework.

cli crawler headless web-spider spider-framework gocrawler

Updated Jan 25, 2023
Go

BruceDone / awesome-crawler

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Dec 20, 2022

yujiosaka / headless-chrome-crawler

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

Updated Nov 11, 2022
JavaScript

Wikipedia: Wikipedia