Skip to content

javahongxi/golab

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
gin
 
 
 
 
 
 
 
 
 
 
 
 
 
 

golab

git clone https://github.com/javahongxi/golab.git
go get ./...

About GoLab

  • lang: Go语法学习与经典示例
  • crawler: 爬虫项目单任务版及并发版
  • crawler_distributed: 爬虫项目分布式版
  • pipeline: 搭建并行处理管道
  • gin: The fastest full-featured web framework

爬虫项目并发版演示

go run mockserver/main.go
  • 启动爬虫
go run crawler/main.go

爬虫项目分布式版演示

go run mockserver/main.go
  • 下载ES并本地启动
bin/elasticsearch
  • 启动存储服务(数据存储到ES)
cd crawler_distributed/persist/server
go run itemsaver.go --port=9090
  • 启动多个worker服务(爬取数据)
cd crawler_distributed/worker/server
go run worker.go --port=9091
go run worker.go --port=9092
  • 启动爬虫
cd crawler_distributed
go run main.go --itemsaver_host=:9090 --worker_hosts=:9091,:9092

并行处理管道演示

示例:对大量数据的外部排序进行并行处理

  • 生成大量数据(写到文件)
go run pipeline/createfile/main.go
  • 演示单机版并行处理
go run pipeline/demo/main.go
  • 演示网络版并行处理
go run pipeline/netdemo/main.go

Go Projects

© hongxi.org | go.hongxi.org

Releases

No releases published

Packages

No packages published

Languages