- scrapy 1.5.1
- airflow 1.10.1
- python 3.5.2
- elasticsearch 6.5.3
- pyspark (spark 2.4.0)
curl -X PUT "localhost:9200/ybvideos" -H 'Content-Type: application/json' -d' { "settings" : { "number_of_shards" : 1 }, "mappings" : { "_doc" : { "properties" : { "snippet":{ "properties": { "channelId" : { "type" : "keyword" }, "title" : {"type" : "text" ,"analyzer": "standard"}, "description" : {"type" : "text" ,"analyzer": "standard"}, "channelTitle": {"type" : "text" ,"analyzer": "standard"}, "publishedAt": {"type" : "date"} } }, "id":{ "properties" : { "videoId":{ "type" : "keyword" } } }, "etag":{"type" : "keyword"} } } } } '
scrapy runspider crawler/spiders/search.py -o ybi.jl
python indexs/writer.py --file ybi.jl
spark-submit analyzers/count.py --file ybi.jl --option word
spark-submit analyzers/count.py --file ybi.jl --option channeltitle
cp dags/pipeline.py ~/airflow/dags/ python ~/airflow/dags/pipeline.py airflow test youtube_crawler scrapy 2018-12-15 airflow test youtube_crawler index 2018-12-15 airflow test youtube_crawler word_count 2018-12-15 airflow test youtube_crawler channel_title_count 2018-12-15 airflow backfill youtube_crawler -s 2018-12-15
- I think just use download_delay is not good enough for throttle,we can set delay less than 10000/3600 (assume 1 hour 10000 is limit) and every hour give crawler 10000 token,we should stop when token use out.
- maybe keyword as a event triggle pipeline.
- check output file when report (for now just print ok)
- transform youtube json format.
- filter datas invalid
- make a classifier using searchword,title ,catetory , channel and description .
- I don't know what is etag in response data. maybe useful.
- download the video.
- security.