Skip to content
A running look at current weather in a historical context.
JavaScript Python CSS
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data-processing
data
dist
images
js
styles
.editorconfig
.gitignore
.jshintrc
Gruntfile.js
LICENSE.txt
README.md
bower.json
bower_map.json
index-build.html
index-deploy.html
index.html
package.json
requirements.txt

README.md

MinnPost Climate

A running look at local, current weather against historical climate patterns.

Unless otherwise noted, MinnPost projects on Github are story-driven and meant for transparency sake and not focused on re-use. For a list of our more reusable projects, go to code.minnpost.com.

Data

Data sources are from the National Oceanic and Atmospheric Administration (NOAA) National Climatic Data Center (NCDC) datasets, as well as the state-level Minnesota Climatology Office which run through the University of Minnesota.

Ultimately what we want is a daily summary for every day (including today) for a specific location (Twin Cities). Outside of getting conditions right now, this should be fairly straightforward, but here is why it's not:

  • The main Twin Cities weather stations switched from downtown Minneapolis to the MSP Int'l Airport in 1938.
  • The GHCN data source (see below) does not seem to have the data for the previous weather station.
  • The GHCN data source is not updated daily, meaning that it lags behind a few days.
  • Historical data sources do not contain today's data.
  • The GSOD data (see below) does not have snow data.

Data sources

  • NOAA Climatological Normals (1981-2010): "Climate Normals are the latest three-decade averages of climatological variables, including temperature and precipitation."
    • What are Normals.
    • Use of Normals. "Meteorologists and climatologists regularly use Normals for placing recent climate conditions into a historical context."
  • "Global Historical Climate Network (GHCN) includes daily observations from around the world. The dataset includes observations from World Meteorological Organization, Cooperative, and CoCoRaHS networks."
    • GHCN does not contain an average temperature, so we use an average of the high and low.
    • GHCN is not up to date and usually lags a few days behind the current date.
    • GHCN list of stations
      • The GHCN Station ID for Minneapolis/St. Paul (MSP) airport is USW00014922. This can be used for the GHCN and Normals sets (though the GSOD identifiers are different).
  • Minnesota Climatology Office Historical Climate Data Listings for the Twin Cities datasets is used for daily data prior to 1938.
  • Preliminary Monthly Climate Data provided by the NOAA National Weather Service (NWS) is used to fill in come gaps for recent history (last couple weeks).
  • Feeds of Current Weather Conditions provided by the NOAA National Weather Service (NWS) is used to get today's observations.
  • Global Surface Summary of Day (GSOD) which is a global collection of recorded conditions each day.
    • GSOD is used to fill in the most recent data.
    • GSOD does contain an average.
    • GSOD does not contain snowfall data.
    • GSOD is updated at leat once a day, but its data should not be used for today.
    • GSOD is used specifically in case we can't get data for recent days through other sources.

Other

Data processing

  • A scraper is written get the historical and recent data. It is meant to be run on the ScraperWiki platform but can be run locally with the following command and will create a local scraperwiki.sqlite database:
    • python data-processing/daily-scraperwiki.py
    • You can query the actual scraper with something like the following: https://premium.scraperwiki.com/d7fssyq/a43576483d6f43a/sql/?q=[[[SQL_QUERY]]]

Development and running locally

Prerequisites

All commands are assumed to on the command line, often called the Terminal, unless otherwise noted. The following will install technologies needed for the other steps and will only needed to be run once on your computer so there is a good chance you already have these technologies on your computer.

  1. Install Git.
    • On a Mac, install Homebrew, then do: brew install git
  2. Install NodeJS.
    • On a Mac, do: brew install node
  3. Install Grunt: npm install -g grunt-cli
  4. Install Bower: npm install -g bower
  5. Install Ruby, though it may already be on your system.
  6. Install Bundler, though it may already be on your system: gem install bundler
  7. Install Compass: gem install compass
    • On a Mac do: sudo gem install compass
  8. Install Python, though it may already be on your system.
  9. Install pip: easy_install pip
  10. (Optional) Use virtualenv, where .env is an environment name that you can change if you want.
    1. easy_install virtualenv
    2. virtualenv .env
    3. cd .env && source bin/activiate; cd -;

Get code and install packages

Get the code for this project and install the necessary dependency libraries and packages.

  1. Check out this code with Git: git clone https://github.com/MinnPost/minnpost-climate.git
  2. Go into the template directory: cd minnpost-climate
  3. Install NodeJS packages: npm install
  4. Install Bower components: bower install
  5. Install Python packages: pip install -r requirements.txt

Running

  1. Run: grunt server
    • This will run a local webserver for development and you can view the application in your web browser at http://localhost:8804.
    • Utilize index.html for development, while index-deploy.html is used for the deployed version, and index-build.html is used to test the build before deployment.
    • The server runs grunt watch which will watch for linting JS files and compiling SASS. If you have your own webserver, feel free to use that with just this command.

Build

To build or compile all the assets together for easy and efficient deployment, do the following. It will create all the files in the dist/ folder.

  1. Run: grunt

Deploy

Deploying will push the relevant files up to Amazon's AWS S3 so that they can be easily referenced on the MinnPost site. This is specific to MinnPost, and your deployment might be different.

  1. Run: grunt deploy

Hacks

List any hacks used in this project, such as forked repos. Link to pull request or repo and issue.

About Us

MinnData, the MinnPost data team, is Alan, Tom, and Kaeti and all the awesome contributors to open source projects we utilize. See our work at minnpost.com/data.


                                               .--.
                                               `.  \
                                                 \  \
                                                  .  \
                                                  :   .
                                                  |    .
                                                  |    :
                                                  |    |
  ..._  ___                                       |    |
 `."".`''''""--..___                              |    |
 ,-\  \             ""-...__         _____________/    |
 / ` " '                    `""""""""                  .
 \                                                      L
 (>                                                      \
/                                                         \
\_    ___..---.                                            L
  `--'         '.                                           \
                 .                                           \_
                _/`.                                           `.._
             .'     -.                                             `.
            /     __.-Y     /''''''-...___,...--------.._            |
           /   _."    |    /                ' .      \   '---..._    |
          /   /      /    /                _,. '    ,/           |   |
          \_,'     _.'   /              /''     _,-'            _|   |
                  '     /               `-----''               /     |
                  `...-'                                       `...-'

You can’t perform that action at this time.