hk0weather is a open source project to scrape useful weather data from Hong Kong Observatory, it is written in python.
With scrapy web scraping framework and regular expression library, collected weather data can be converted to machine-readable formats (eg. JSON).
With django web framework, collected weather data will be stored in django, and accessible through django web admin UI. sqlite3 is default database file format, and it can be connected with MySQL and other database systems supported by django.
Source code is available on github.
$ virtualenv hk0weatherenv
$ source hk0weatherenv/bin/activate
$ pip install scrapy
$ pip install django
$ git clone https://github.com/sammyfung/hk0weather.git
$ cd hk0data
$ python manage.py syncdb
Running a Django CMS (with web admin UI)
$ cd hk0data
$ python manage.py runserver &
Django web admin UI can be access at: http://localhost:8000/admin
Run a scrapy web scraper
To run a scrapy web scraper.
$ scrapy crawl
To run a scrapy web scraper with output file in json format.
$ scrapy crawl -o output_filename -t json
List of Spiders
- hko9dayforecast (under development): Hong Kong 9 day Weather Report from HKO.
- hkocurrwx: Current Hourly Hong Kong Weather Report from HKO.
- hkoforecast: Hong Kong Next 24 hour Weather Forecast Report from HKO.
- hkrainfall: Hong Kong Rainfall Data (Hourly update) from HKO.
- regionalwx: Hong Kong Regional Weather Data (10-min update) from HKO.
For general discussion of hk0weather project, please go to hk0weather google group. Please feel freely to ask questions or post your suggestions / comments.
I introduced this project on my following Chinese blog.
And I also presented it at BarCampHK 2013 and a local open source workshop, hereby is my slide.