Skip to content
master
Go to file
Code

Latest commit

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
gin
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

golab

git clone https://github.com/javahongxi/golab.git
go get ./...

About GoLab

  • lang: Go语法学习与经典示例
  • crawler: 爬虫项目单任务版及并发版
  • crawler_distributed: 爬虫项目分布式版
  • pipeline: 搭建并行处理管道

爬虫项目并发版演示

go run mockserver/main.go
  • 启动爬虫
go run crawler/main.go

爬虫项目分布式版演示

go run mockserver/main.go
  • 下载ES并本地启动
bin/elasticsearch
  • 启动存储服务(数据存储到ES)
cd crawler_distributed/persist/server
go run itemsaver.go --port=9090
  • 启动多个worker服务(爬取数据)
cd crawler_distributed/worker/server
go run worker.go --port=9091
go run worker.go --port=9092
  • 启动爬虫
cd crawler_distributed
go run main.go --itemsaver_host=:9090 --worker_hosts=:9091,:9092

并行处理管道演示

示例:对大量数据的外部排序进行并行处理

  • 生成大量数据(写到文件)
go run pipeline/createfile/main.go
  • 演示单机版并行处理
go run pipeline/demo/main.go
  • 演示网络版并行处理
go run pipeline/netdemo/main.go

Go Projects

© hongxi.org | go.hongxi.org

About

Go 生态研究 🔥🔥🔥🔥🔥

Topics

Resources

Releases

No releases published

Packages

No packages published

Languages