Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
-
Updated
Sep 11, 2024 - Go
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middleware and supports various parsing and storage methods. Additionally, it supports distributed deployment. 基于golang实现的爬虫框架,编写简单,性能强劲。内置了丰富的实用中间件,支持多种解析、保存方式,支持分布式部署。
GoScrapy: Harnessing Go's power for blazingly fast web scraping, inspired by Python's Scrapy framework.
a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
GoCrawler is a web crawling framework written in Go, inspired by Scrapy. Support this project at https://ko-fi.com/gocrawl
A simple crawler library implement with Golang
go-crawling-bazaar(gcb) is a distributed crawling market based on Blockchain technology.
Add a description, image, and links to the scrapy topic page so that developers can more easily learn about it.
To associate your repository with the scrapy topic, visit your repo's landing page and select "manage topics."