Skip to content
This repository

Open Data Census website and dashboard

Merge pull request #316 from okfn/feature/allow-simple-submission

Always "canReview" on submission, which allows simple submission
latest commit 1aad54935d
Rufus Pollock rgrp authored
Octocat-spinner-32 bin [#315,cli][s]: command to add existing apps as heroku/git remotes loc… April 13, 2014
Octocat-spinner-32 content [faq][xs]: remove out of date section. April 01, 2014
Octocat-spinner-32 doc [#315,cli][s]: command to add existing apps as heroku/git remotes loc… April 13, 2014
Octocat-spinner-32 lib [#315,cli][s]: command to add existing apps as heroku/git remotes loc… April 13, 2014
Octocat-spinner-32 locale Generate message files February 21, 2014
Octocat-spinner-32 public writing mode confuses IE February 24, 2014
Octocat-spinner-32 routes Always "canReview" on submission, which allows simple submission April 14, 2014
Octocat-spinner-32 templates Whitespace changes for tests February 26, 2014
Octocat-spinner-32 tests Merge pull request #306 from okfn/feature/281 April 01, 2014
Octocat-spinner-32 utils aaah fixed the sparql query - don't count all the changes as seperate September 14, 2012
Octocat-spinner-32 .gitignore [#262,deploy][m]: working script for setting up custom censuses. February 12, 2014
Octocat-spinner-32 .gitmodules Major JavaScript cleanup October 23, 2013
Octocat-spinner-32 .jshintrc Override global .jshintrc with project specifics February 21, 2014
Octocat-spinner-32 .travis.yml Set correct credentials for Travis. March 27, 2014
Octocat-spinner-32 CHANGES.md [#315,cli][s]: command to add existing apps as heroku/git remotes loc… April 13, 2014
Octocat-spinner-32 Procfile [refactor][s]: move running of app to run.js to avoid running the app… February 03, 2014
Octocat-spinner-32 README.md [README][xs]: reference demo site. April 01, 2014
Octocat-spinner-32 app.js Post now to /submission/:id February 26, 2014
Octocat-spinner-32 generate-translations.sh Generate message files February 21, 2014
Octocat-spinner-32 package.json Switch to main google-spreadsheet module March 14, 2014
Octocat-spinner-32 process.py [data,fix][xs]: correct for typo in dataset name. April 30, 2012
Octocat-spinner-32 run.js [bugfix][xs]: last commit was broken!. February 13, 2014
README.md

Open Data Census

Build Status

Webapp for doing [Open Data Censuses][] including submission workflow, presentation of results and some visualization.

This also includes various ancillary information providing an overview of what is happening with release of open government data around the world (and initiatives related to it).

Demo Site

If you want to check out what an Open Data Census site looks like we have a demo site running at:

http://demo.census.okfn.org/

Overview

See doc/admin.md

Developer Stuff

The app is a simple Express NodeJS app designed to be deployed on Heroku.

Config boot sequence:

  • App boots and looks up local config (set by census deployer)
  • Looks up environment variable CONFIG_URL (plus sensitive config like DB login)
  • Loads CSV file at CONFIG_URL - this file has pointers to all other config information (see below for a template)
  • Loads all other config CSV files (Places, Datasets, Questions)

Auth

For user Auth we use Google and their OAuth 2.0.

you will need to create an App on Facebook developers section and set various config. See config section below for detail.


Developing the Code

Install Locally

To install do the following:

  1. Get the code and init the submodules (for recline vendor library)

    git clone https://github.com/okfn/opendatacensus
    git submodule init
    git submodule update
    
  2. Install node dependencies

    cd opendatacensus
    npm install .
    
  3. Run the app

    node run.js
    
  4. Should now be running at http://localhost:5000

Configuration

Core configuration is listed in lib/config.js which loads from environment variables and then via lib/util.js load method to pull in config from CSV files.

Over-riding for development

For convenience when doing local development, you can selectively override your own local config using a settings.json as follows:

  • Create settings.json
  • Copy the config object from lib/config.js and override relevant parts. Note you don't need the whole object only the bits you want to change. For example:

      {
        "google": {
          "user": "xxx",
          "password": "yyy"
        }
      }
    

Note this will not work for Heroku - instead you need to do everything via environment variables: https://devcenter.heroku.com/articles/config-vars

i18n For Templates

When templates change, the translations have to be changed. Extract the files by running this command:

./node_modules/.bin/extract-pot --output=../messages.pot --locale locale -t jinja -f html templates

You will need the GNU gettext commands. See here for more information.

To generate the .po files and compile them to json, run the following command:

./generate-translations.sh

To update the list of languages, modify line #4 to add more in the for loop.

i18n For Config

Any column can be internationalised by adding another column with @locale after it. For example, the description column can be translated to German by adding a column of description@de. Only languages which have template translations created for them are valid. The locales setting in the config document can be used to restrict the number of locales available. The first locale in the list is the default locale.

Running Tests

  • Install dev dependencies and mocha - npm install -d
  • Get the opendatacensustest google user login and add to settings.json

Then run the tests:

mocha tests/

Heroku Deployment

We have multiple apps on Heroku including:

  • Production: opendatacensus - push there from production branch
  • Staging: opendatacensus-staging - push from master

To work with a given remote:

heroku --remote production ...

To work with these do:

heroku git:remote -r production -a opendatacensus
heroku git:remote -r staging -a opendatacensus-staging
# this way git push heroku master will push to staging
heroku git:remote -a opendatacensus-staging

To avoid error suggest making the staging app the default:

git config heroku.remote staging

Appendix - Why Google Spreadsheets for the DB

Pros

  • being easy to hand-edit and view (esp for non-techies)
  • multiple formats
  • versioned (so all changes are recorded)

Cons

  • Google Spreadsheets has limited storage (400k cells etc). However, our data requirements are usually quite limited for each census.
Something went wrong with that request. Please try again.