Project Open Data Dashboard
PHP HTML Other
Permalink
Failed to load latest commit information.
application Display full S3 URL in logs Feb 16, 2017
archive New directory structure for archives Jun 20, 2014
css disclaimer margins Dec 13, 2016
digital-strategy adding glyphicons-halflings-regular.woff2 Jan 9, 2017
documentation couple more typo fixes Jun 22, 2016
downloads Option to archive data.json files Feb 24, 2014
fonts adding glyphicons-halflings-regular.woff2 Jan 9, 2017
js Autofill automated metrics where available Dec 16, 2015
schema Support FITARA schema for CSV conversions Aug 28, 2015
sql Update SQL schema Jun 19, 2014
system core fix :: referrence returned Aug 12, 2016
user_guide Initial commit with Codeigniter 2.1.4 Oct 24, 2013
.gitignore merge upstream changes Nov 22, 2016
.gitmodules remove digital-strategy module Nov 16, 2016
CONTRIBUTING.md Create CONTRIBUTING.md Oct 27, 2015
LICENSE.md Rename license.md to LICENSE.md Oct 27, 2015
README.md Merge pull request #95 from md5/patch-2 Sep 10, 2015
apple-touch-icon-114x114-precomposed.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
apple-touch-icon-144x144-precomposed.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
apple-touch-icon-57x57-precomposed.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
apple-touch-icon-72x72-precomposed.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
apple-touch-icon-precomposed.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
apple-touch-icon.png Fresh frontend using intializer for bootstrap, modernizr, jquery, etc Oct 25, 2013
composer.json JSON Pretty print Jan 3, 2017
composer.lock Cleanup on exception Feb 16, 2017
favicon.ico updating favicon Nov 15, 2013
index.php.sample Don't track index.php since it's configurable Jun 17, 2014
sample.htaccess adding additional folder paths May 27, 2015

README.md

The Project Open Data Dashboard provides a variety of tools and capabilities to help manage the implementation of Project Open Data. It is primary used for Federal agencies, but also provides tools and resources for use by other entities like state and local government.

The primary place for the user-facing documentation is http://data.civicagency.org/docs

Federal agencies were seeded using the USA.gov Federal Agency Directory API and the IDs provided by that resource are used as the primary IDs on this dashboard.

Features

  • Dashboard overview of the status of each federal agency's implementation of Project Open Data for each milestone.
  • Permissioned Content Editing for the fields in the dashboard that can't be automated. The fields are stored as JSON objects so the data model is very flexible and can be customized without database changes. User accounts are handled via Github.
  • Automated crawls for each agency to report metrics from Project Open Data assets (data.json, digitalstrategy.json, /data page, etc). This includes reporting on the number of datasets and validation against the Project Open Data metadata schema.
  • A validator to validate Project Open Data data.json files via URL, file upload, or text input. This can be used for testing both data.json Public Data Listing files as well as the Enterprise Data Inventory JSON. The validator can be used both by Federal agencies as well as non-federal entities by specifiying the Non-Federal schema.
  • Converters to transform a CSV into a data.json file or to export existing data from Data.gov
  • Changeset viewer to see how a data.json file for an agency compares to metadata currently hosted on data.gov

CLI Interface

In addition to the web interface, there's also a Command Line Interface to manage the crawls of data.json, digitalstrategy.json, and /data pages. This is helpful to run specific updates, but it's primary use is with a CRON job.

From the root of the application, you can update the status of agencies using a few different options on the campaign controller. The syntax is:

$ php index.php campaign status [id] [component]

If you wanted to update all components (data.json, digitalstrategy.json, /data) for all agencies, you'd run this command:

$ php index.php campaign status all all

If you just wanted to update the data.json status for CFO Act agencies you'd run:

$ php index.php campaign status cfo-act datajson

If you just wanted to update the digitalstrategy.json status for the Department of Agriculture you'd run:

$ php index.php campaign status 49015 digitalstrategy

The options for [id] are: all,cfo-act, or the ID provided by the USA.gov Federal Agency Directory API

The options for [component] are: all, datajson, datapage, digitalstrategy, download.

  • The datajson component captures the basic charactersitics of a request to an agency's data.json file (like whether it returns an HTTP 200) and then attempts to parse the file, validate against the schema, and provide other reporting metrics like the number of datasets listed.
  • The digitalstrategy component captures the basic charactersitics of a request to an agency's digitalstrategy.json file (like whether it returns an HTTP 200)
  • The datapage component captures the basic charactersitics of a request to an agency's /data page (like whether it returns an HTTP 200)
  • The download component downloads an archive of the data.json and digitalstrategy.json files
  • As you'd expect, all does all of these things at once.

Installation

This is a CodeIgniter PHP application. Installation primarily consists of editing config files in /application/config and importing the database schema.

  1. Grab the code git clone https://github.com/GSA/project-open-data-dashboard.git
  2. Copy /sample.htaccess to /.htaccess. You may need to adjust the configuration of your .htaccess file to match your environment.
  3. Copy /index.php.sample to /index.php. The index.php file is where you specify whether you're running a production or development environemtn. If you are running a production environment, you'll need to edit line 21 to define('ENVIRONMENT', 'production'); otherwise you can leave the file unchanged.
    • Note that wherever you see <env> in the following configuration steps, you should replace that with the environment you're currently using. To begin with, you'll probably want this to be: development
  4. Copy /application/config/config.php.sample to /application/config/<env>/config.php
  5. Edit /application/config/<env>/config.php with:
    • The base_url where this application will run. This actually shouldn't required for anything other than the following github_oauth_redirect which depends on it, but you can set that manually if you'd like.
    • Your Github OAuth Client ID, Client Secret, and redirect URL for GitHub authentication
    • The path to where you want to store the archival json files (archive_dir) - give this directory adequate permissions for your server to write to
    • You can pre-approave github users to have admin rights by adding their github username to the array in the pre_approved_admins option
    • Set an encryption key for the encryption_key option (around line 253). You can generate a key with a command like LANG=C tr -dc A-Za-z0-9_ < /dev/urandom | head -c 32 | xargs or get one from this generator
  6. Copy /application/config/upload.php.sample to /application/config/<env>/upload.php
  7. Edit /application/config/<env>/upload.php with the upload_path where you want to save uploaded CSV files (give this directory adequate permissions for your server to write to)
  8. Create a local database and import the SQL database tables found in /sql/ into this local database:
    • mysql> CREATE DATABASE pod_dashboard;
    • $> mysql -u root -p pod_dashboard < datagov_campaign.sql
    • $> mysql -u root -p pod_dashboard < offices.sql
    • $> mysql -u root -p pod_dashboard < notes.sql
    • $> mysql -u root -p pod_dashboard < users_auth.sql
  9. Copy /application/config/database.php.sample to /application/config/<env>/database.php
  10. Edit /application/config/<env>/database.php with your local database settings

If you have an empty database offices table in the database, you'll also want to seed it with agency data by running the import script (/application/controllers/import.php) from a command line. You'll also need to temporarily change the import_active option in config.php to true

First populate the top of the agency hierarchy:

$ php index.php import

Second, populate all the subagencies:

$ php index.php import children

Limitations

Currently this tool does not handle large files in a memory efficient way. If you are unable to utilize a high amount of memory and are at risk of timeouts, you should set the maximum file size that the application can handle so it will avoid large files and fail more gracefully. The maximum size of JSON files to parse can be set with the max_remote_size option in config.php