Skip to content
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
97 lines (67 sloc) 4.21 KB

Civic Pulse

About the project

Civic Pulse is a project from MuckRock and Code for Boston. We scan thousands of government websites to check how well they stack up on security, accessibility, and public accountability. You can join us every Tuesday night to help out.

A screenshot of what a Civic Scorecard looks like

Getting started basics

Project goals

The goal is to create an automatically updated database that tracks, over time, how well government agencies websites at the state, local, and federal levels follow best practices when it comes to HTTPS security, mobile friendliness, reader accessibility, and other key areas.

Over time, we hope to show whether both individual agencies are improving or worsening, as well as help highlight national shifts along the metrics we monitor. Individual pages show the most recent snapshot ranking, but our API will make historical data available.

Current status

The project is currently in testing stages, as we work to both develop usable, accurate data and build a pipeline for regularly populating it. The site currently can run locally, but several of the data categories are filled with randomized testing data and any report cards generated are for demonstration purposes only. These scores do not represent actual scores for agencies.

Installation instructions

Clone the repository:

git clone

Navigate to the base directory of the reposistory and prepare to install depedencies.

To start, it is recommend to create a virtual environment. If you have not used virtualenv before, install it with: pip install virtualenv.

# Create a virtual environment to manage dependencies
virtualenv venv
source venv/bin/activate

Now install the dependencies with pip:

# Install requirements.txt
pip install -r requirements.txt

After the dependencies have installed, we want to prepare the database.

# Perform data migrations
python migrate

Then, we need to import a CSV file containing existing agency information. Start by running a Django shell:

python shell

# From within the shell
>>> from apps.civic_pulse.utils.load_models import *
>>> fill_agency_objects()
>>> exit()

create user for the scraper.

This step is needed in order to connect the api with the scrapers. If you do not wish to do that, then this may be skipped. We need to create a dummy user for the scraper to be able to access the api. The api is part of the Django projet. Note: The scrapers live in an independent environment not neccessarily in the same server as the Django website. The scrapers read and write data to the website using api endpoints. To create a token:

  • create an admin user to be able to login to the admin portal of the website: /admin
  python createsuperuser --username admin --email
  # enter the password when prompted. It can be any password that you wish to use. 
  # It is used for login to the admin website.
  • login to the admin website and create a user for the scraper.
  • create a token for the scraper user using the following command
./ drf_create_token <username>

Finally, the database is ready to go! We are now ready to run the server:

python runserver

Navigate in your browser to and you should see a list of agencies.

You can’t perform that action at this time.