A running look at local, current weather against historical climate patterns.
Unless otherwise noted, MinnPost projects on Github are story-driven and meant for transparency sake and not focused on re-use. For a list of our more reusable projects, go to code.minnpost.com.
Data sources are from the National Oceanic and Atmospheric Administration (NOAA) National Climatic Data Center (NCDC) datasets, as well as the state-level Minnesota Climatology Office which run through the University of Minnesota.
Ultimately what we want is a daily summary for every day (including today) for a specific location (Twin Cities). Outside of getting conditions right now, this should be fairly straightforward, but here is why it's not:
- The main Twin Cities weather stations switched from downtown Minneapolis to the MSP Int'l Airport in 1938.
- The GHCN data source (see below) does not seem to have the data for the previous weather station.
- The GHCN data source is not updated daily, meaning that it lags behind a few days.
- Historical data sources do not contain today's data.
- The GSOD data (see below) does not have snow data.
- NOAA Climatological Normals (1981-2010): "Climate Normals are the latest three-decade averages of climatological variables, including temperature and precipitation."
- What are Normals.
- Use of Normals. "Meteorologists and climatologists regularly use Normals for placing recent climate conditions into a historical context."
- "Global Historical Climate Network (GHCN) includes daily observations from around the world. The dataset includes observations from World Meteorological Organization, Cooperative, and CoCoRaHS networks."
- GHCN does not contain an average temperature, so we use an average of the high and low.
- GHCN is not up to date and usually lags a few days behind the current date.
- GHCN list of stations
- The GHCN Station ID for Minneapolis/St. Paul (MSP) airport is USW00014922. This can be used for the GHCN and Normals sets (though the GSOD identifiers are different).
- Minnesota Climatology Office Historical Climate Data Listings for the Twin Cities datasets is used for daily data prior to 1938.
- Preliminary Monthly Climate Data provided by the NOAA National Weather Service (NWS) is used to fill in come gaps for recent history (last couple weeks).
- Feeds of Current Weather Conditions provided by the NOAA National Weather Service (NWS) is used to get today's observations.
- Global Surface Summary of Day (GSOD) which is a global collection of recorded conditions each day.
- GSOD is used to fill in the most recent data.
- GSOD does contain an average.
- GSOD does not contain snowfall data.
- GSOD is updated at leat once a day, but its data should not be used for today.
- GSOD is used specifically in case we can't get data for recent days through other sources.
- The CLIMATE tool should have combined data: http://mrcc.isws.illinois.edu/CLIMATE/.
- For long term record checking, see http://threadex.rcc-acis.org/.
- A scraper is written get the historical and recent data. It is meant to be run on the ScraperWiki platform but can be run locally with the following command and will create a local
scraperwiki.sqlitedatabase:python data-processing/daily-scraperwiki.py- You can query the actual scraper with something like the following:
https://premium.scraperwiki.com/d7fssyq/a43576483d6f43a/sql/?q=[[[SQL_QUERY]]]
All commands are assumed to on the command line, often called the Terminal, unless otherwise noted. The following will install technologies needed for the other steps and will only needed to be run once on your computer so there is a good chance you already have these technologies on your computer.
- Install Git.
- On a Mac, install Homebrew, then do:
brew install git
- On a Mac, install Homebrew, then do:
- Install NodeJS.
- On a Mac, do:
brew install node
- On a Mac, do:
- Install Grunt:
npm install -g grunt-cli - Install Bower:
npm install -g bower - Install Ruby, though it may already be on your system.
- Install Bundler, though it may already be on your system:
gem install bundler - Install Compass:
gem install compass- On a Mac do:
sudo gem install compass
- On a Mac do:
- Install Python, though it may already be on your system.
- Install pip:
easy_install pip - (Optional) Use virtualenv, where
.envis an environment name that you can change if you want.easy_install virtualenvvirtualenv .envcd .env && source bin/activiate; cd -;
Get the code for this project and install the necessary dependency libraries and packages.
- Check out this code with Git:
git clone https://github.com/MinnPost/minnpost-climate.git - Go into the template directory:
cd minnpost-climate - Install NodeJS packages:
npm install - Install Bower components:
bower install - Install Python packages:
pip install -r requirements.txt
- Run:
grunt server- This will run a local webserver for development and you can view the application in your web browser at http://localhost:8804.
- Utilize
index.htmlfor development, whileindex-deploy.htmlis used for the deployed version, andindex-build.htmlis used to test the build before deployment. - The server runs
grunt watchwhich will watch for linting JS files and compiling SASS. If you have your own webserver, feel free to use that with just this command.
To build or compile all the assets together for easy and efficient deployment, do the following. It will create all the files in the dist/ folder.
- Run:
grunt
Deploying will push the relevant files up to Amazon's AWS S3 so that they can be easily referenced on the MinnPost site. This is specific to MinnPost, and your deployment might be different.
- Run:
grunt deploy
List any hacks used in this project, such as forked repos. Link to pull request or repo and issue.
MinnData, the MinnPost data team, is Alan, Tom, and Kaeti and all the awesome contributors to open source projects we utilize. See our work at minnpost.com/data.
.--.
`. \
\ \
. \
: .
| .
| :
| |
..._ ___ | |
`."".`''''""--..___ | |
,-\ \ ""-...__ _____________/ |
/ ` " ' `"""""""" .
\ L
(> \
/ \
\_ ___..---. L
`--' '. \
. \_
_/`. `.._
.' -. `.
/ __.-Y /''''''-...___,...--------.._ |
/ _." | / ' . \ '---..._ |
/ / / / _,. ' ,/ | |
\_,' _.' / /'' _,-' _| |
' / `-----'' / |
`...-' `...-'