Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Build Status codecov

hk0weather

hk0weather is an open source web scraper project using Scrapy to collect the useful weather data from Hong Kong Observatory website.

Scrapy can output collected weather data into the machine-readable formats (eg. CSV, JSON, XML).

Optionally, this project supports a Django app 'openweather' to store the collected weather data to Django web framework, and the data can be shown on web through the Django admin UI.

Available Spiders

  1. regional: Hong Kong Regional Weather Data in 10-minutes update from HKO.
  2. rainfall: Hong Kong Rainfall Data in hourly update from HKO.
  3. hkoforecast: Hong Kong Next 24 hour Weather Forecast Report from HKO Open Data.
  4. hko9dayforecast: Hong Kong 9-day Weather Report from HKO Open Data.

Installation Example

  1. Cloning and setup hk0weather in a Py3 virtual environment

    git clone https://github.com/sammyfung/hk0weather.git  
    virtualenv hk0weatherenv  
    source hk0weatherenv/bin/activate  
    cd hk0weather   
    pip install -r requirements.txt    
    
  2. Optional: Setup hk0weather to use openweather

    pip install -r requirements-django.txt    
    cd ..   
    django-admin startproject yourweatherproject   
    cd yourweatherproject   
    git clone https://github.com/sammyfung/openweather.git   
    

    Please add 'openweather' to INSTALLED_APPS in Django yourweatherproject/settings.py.

    ./manage.py makemigrations    
    ./manage.py migrate   
    ./manage.py createsuperuser   
    ./manage.py runserver &    
    cd ../hk0weather     
    

    Django daemon is now running in the background, its web admin UI can be access at http://localhost:8000/admin.

    export PYTHONPATH=/your-full-path-to/yourweatherproject    
    export DJANGO_SETTINGS_MODULE=yourweatherproject.settings   
    

    Please export PYTHONPATH and DJANGO_SETTINGS_MODULE again after every activation of the Py3 virtual environment.

Run a Scrapy spider

Activate the Py3 virtual environment once before the first running of web spiders.

source hk0weatherenv/bin/activate  

Optionally, if Django is in use, export PYTHONPATH and DJANGO_SETTINGS_MODULE.

export PYTHONPATH=/your-full-path-to/yourweatherproject    
export DJANGO_SETTINGS_MODULE=yourweatherproject.settings   

Optionally, list all available spiders.

scrapy list 

Run a specific spider (eg. regional) in Scrapy

scrapy crawl regional   

and optionally use -t (file format) and -o (filename) to output the data in a json file.

scrapy crawl regional -t json -o test.json

Sponsors

Calvin Tsang.

Thanks for my sponsors, please consider to sponsor my works.

References

About

Web scraper project to collect the useful Hong Kong weather data from HKO website

Topics

Resources

Releases

No releases published

Sponsor this project

 

Packages

No packages published