Skip to content

cool-firer/WeatherCrawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 

Repository files navigation

WeatherCrawler

爬取天气数据的爬虫,会抓取从今天之后的七天天气数据

运行之前,确保已经安装scrapy、mongodb

安装scrapy: pip install scrapy

运行爬虫

scrapy crawl province_spider

或者指定日志级别的方式启动:

scrapy crawl province_spider --loglevel INFO

会依次从各省、市、县爬取天气数据,存入数据库,最终的数据如下:


weather> db.wea.findOne()
{
        "_id" : ObjectId("598c1a0efb64421dcc437c47"),
        "province" : "湖北",
        "city" : "武汉",
        "data" : [
                {
                        "wind_force" : "微风",
                        "max_tem" : "35",
                        "min_tem" : "27℃",
                        "wind_direction" : [
                                "无持续风向",
                                "无持续风向"
                        ],
                        "day" : "10日(今天)",
                        "desc" : "晴转多云"
                },
              ...
  }

About

爬取天气数据的爬虫

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Languages