golang light-weight image crawler
Go
Latest commit c37a053 Jan 16, 2017 @songtianyi remove debug
Permalink
Failed to load latest commit information.
downloader use rrframework/storage instead Jan 10, 2017
spider spider refinement Jan 16, 2017
.gitignore add jav.go Jan 11, 2017
.travis.yml add jav.go Jan 11, 2017
LICENSE Initial commit Dec 2, 2016
README.md Update README.md Jan 16, 2017
jav.go spider refinement Jan 16, 2017
mzitu.go remove debug Jan 16, 2017
mzituzp.go run gofmt Jan 16, 2017

README.md

laosj(老司机)

Build Status Go Report Card codebeat badge License

logo

基于goquery的轻量级爬虫, 图片下载器支持分布式, 可以自定义下载并发量。

Releases

  • mzitu 爬取mzitu.com/taiwan (图片下载依赖redis作为url队列)
  • mzituzp 爬取mzitu.com/share (图片下载依赖redis作为url队列)
  • jav 获取番号列表及根据番号获取种子文件 (无依赖)

代码上手

下载
go get -u -v github.com/songtianyi/laosj
安装redis
golang.org/x依赖安装
mkdir $GOPATH/src/golang.org/x
cd $GOPATH/src/golang.org/x
git clone https://github.com/golang/net.git
编译并运行样例
go build mzituzp.go
./mzituzp

截图

laosj