ScrapySite

ScrapySite

how build

git clone git@github.com:hktalent/scrapysite.git
cd scrapysite
go build main.go
#or build for all palteform
make all -f Makefile.cross-compiles
ls -lah release/
# or build 
make all
ls -lah bin/
ls -lah main

how use Elasticsearch

http://127.0.0.1:9200/_cat/indices?v 1、create index

./tools/CreateEs.sh scrapy

http://127.0.0.1:9200/scrapy_index/_doc/

how use

./main -url="http://www.xxx1.cn;http://www.xx2.cn" -resUrl="http://127.0.0.1:9200/st_index/_doc/"

http://127.0.0.1:9200/st_index/_search?q=edu&pretty=true