A dashboard of various metrics, generated nightly from IATI data
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
static Wrapped emails in a tags in FAQ page #515 Sep 17, 2018
.gitignore Move username/password out of git Feb 27, 2014
CHECKLIST.rst Replace all mentions of nightly updates May 8, 2018
CONTRIBUTING.rst Remove Github pages from Dashboard Jan 4, 2017
GPL.md Add GPL license Dec 12, 2013
README.rst Merge pull request #500 from IATI/update-header-link-text Jul 31, 2018
TODO Ensure the dashboard displays 2.01 data successfully Jan 27, 2015
common.py Move retrieval of publisher org type into a seperate function Nov 26, 2015
comprehensiveness.py Merge with master, resolving merge conflicts Feb 3, 2017
config.py.example Move username/password out of git Feb 27, 2014
coverage.py 2015 -> 2016 Sep 25, 2017
data.py Don’t cache JSONDirs recursively Jun 15, 2018
dfi_publishers.csv [#387] Amend coverage methodology for DFI institutions Mar 30, 2016
fetch_data.py Fetch big pages from registry API Jun 6, 2018
fetch_data.sh Update fetch_data.sh Jan 25, 2018
forwardlooking.py [#403] Amend publisher statistics calculations to round to the neares… May 24, 2016
get_stats.sh Reduce verbosity of get_stats.sh Feb 2, 2017
git.sh Add timestamps to log messages Aug 8, 2018
humanitarian.py Cast humanitarian publisher stats denominators to float values Dec 4, 2017
licenses.py [#408] Fix license bug if publishers have no data URL Sep 30, 2016
make_csv.py [#370] Remove references to Data Tickets across the Dashboard Jul 3, 2017
make_html.py Fix link to organisation standard pages Jul 17, 2018
plots.py Make plots.py significantly faster Jun 6, 2018
registry_id_relationships.csv add canada publisher-id mapping Jan 27, 2016
requirements.txt Merge with master, resolving merge conflicts Feb 3, 2017
requirements_dev.txt [requires.io] dependency update Jan 22, 2017
speakers_kit.py [#235] Pull both 1.05 and 2.01 codelists and use as appropriate Jan 22, 2015
summary_stats.py Rename summary_statistics -> summary_stats Feb 23, 2017
text.py Update Home link for clarity under new branding Jun 29, 2018
timeliness.py [#353][#358] Robust fix for publishers who change their registry ID, … Jan 20, 2016
vars.py Update vars.py Jan 25, 2018

README.rst

IATI Dashboard

https://travis-ci.org/IATI/IATI-Dashboard.svg?branch=master Requirements Status https://coveralls.io/repos/IATI/IATI-Dashboard/badge.png?branch=master

Introduction

The IATI Dashboard is a displays key numbers and graphs about the data on the IATI Registry.

See the Dashboard in action at http://dashboard.iatistandard.org

The Dashboard is in beta, all contents/urls/machine readable downloads are subject to change.

This repository is the code for the Dashboard frontend. Stats are generated from the Registry by stats code in a separate repository - https://github.com/IATI/IATI-Stats

Technology Overview

The dashboard mostly in Python, with some helper Bash scripts.

Python scripts:

  • make_html.py contains is a Flask application that makes use of Frozen Flask to generate some static HTML.
  • make_csv.py generates CSV files.
  • plots.py generates static images of graphs using matplotlib.

Bash helper scripts:

  • The main source of data for the Dashboard is stats generated by IATI-Stats (about the data on the IATI Registry). get_stats.sh can be used to fetch a recently calculated copy of these stats. (or see calculating your own stats section below)
  • The Dashboard also uses various other data from online sources (including GitHub). These can be fetched using fetch_data.sh.
  • git.sh runs all the above commands, see Usage below.

Installation

Requirements:

  • Unix based setup (e.g. Linux, Mac OS X) with bash etc.

  • wget and curl installed

  • Python 2.7

  • Python dev library python-dev

  • python-virtualenv (optional)

  • Development files for libfreetype, libpng, libxml and libxslt e.g. libfreetype6-dev libpng-dev libxml2-dev libxslt-dev.

    (alternatively, you may be able to install some of the python dependencies in requirements.txt using your package manager)

To install:

## Get the code
git clone https://github.com/IATI/IATI-Dashboard.git
cd IATI-Dashboard

## Set up a virtual environment (recommended)
# Create a virtual environment
virtualenv pyenv
# Activate the virtual environment
# (you need to this every time you open a new terminal session)
source pyenv/bin/activate

## Install python dependencies
## Use pip as described below, or your distro's package manager to install
## the dependcies in requirements.txt
# If you are running a less recent linux distro, you will need to install distribute
easy_install -U distribute
pip install -r requirements.txt

# Create a configuration file
# Currently this is only used for GitHub credentials to make requests to the GitHub api
# You should be able to run the dashboard in live development mode (see Development below)
# without this file. (Although the GitHub tab will throw an error).
cp config.py.example config.py # And provide the necessary values

Usage

The following steps are performed routinely on our deployed Dashboard. (On our servers, the Dashboard is actually deployed using this salt file.

# Fetch the necessary calculated stats
./get_stats.sh
# Fetch some extra data from github and github gists
./fetch_data.sh

mkdir out
python plots.py
python make_csv.py
python make_html.py

make_html.py will output a MissingURLGeneratorWarning. This is expected, as some of the URLs defined are for the live development server only (see below).

The full list of steps for our deployment can be found in git.sh. (The name of this is now a misnomer as the output is no longer a git repository - previously a commit was pushed to GitHub pages.)

Development

For development, you can use the live Flask development server, instead of Frozen Flask.

python make_html.py --live

Using the live development server is highly recommended, because it displays full bracktraces for 500 errors, whereas frozen flask does not.

Calculating your own stats for the dashboard

The Dashboard requires a stats-calculated directory, which can be downloaded using the get_stats.sh shell script as described above, or calculated yourself using http://github.com/IATI/IATI-Stats . stats-calculated corresponds to the gitout directory generated by IATI-Stat's git.sh.

Often you only want to regenerate the current stats, use get_stats.sh to download the pre-calculated historical stats and just replace the stats-calculated/current directory with the out directory produced by running the loop, aggregate and invert commands individually.

License

Copyright (C) 2013-2015 Ben Webb <bjwebb67@googlemail.com>
Copyright (C) 2013-2014 David Carpenter <caprenter@gmail.com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.