The US federal government's web traffic.
Clone or download
tdlowden Merge pull request #540 from 18F/revert-539-revert-534-519-report-issue
Revert - put back the footer "Revert "519 report issue""
Latest commit 2136756 Nov 20, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci update circle cmd for tests Nov 7, 2018
.github add issue templates Nov 1, 2018
_agencies Remove data urls in favor of determining everything from slugs Jan 17, 2017
_data_pages Remove data urls in favor of determining everything from slugs Jan 17, 2017
_includes Revert "Revert "519 report issue"" Nov 20, 2018
_layouts Add a developer page to explain the API usage Apr 7, 2017
assets send prod bundle Nov 19, 2018
bin Add CircleCI in the right spot Mar 22, 2018
compliance Create component.yaml Dec 8, 2016
css Revert "Revert "519 report issue"" Nov 20, 2018
docs Update procedures.md Aug 21, 2018
fake-data/live Finalize cloud.gov migration and restore data links (#408) Nov 29, 2016
fonts inline the google font Sep 22, 2015
images Revert "Revert "519 report issue"" Nov 20, 2018
js change text when aria tabs selected Nov 14, 2018
sass added location data, minor visualization bug still needs to be fixed Oct 16, 2015
spec add transformer tests Nov 14, 2018
.about.yml adding more info Oct 17, 2016
.babelrc add formatter tests Nov 7, 2018
.codeclimate.yml Add codeclimate and ESList configs Jan 19, 2017
.dockerignore Update production Docker image to use custom nginx config May 3, 2017
.eslintrc move all js deps to npm and rm uswds js Oct 25, 2018
.gitignore add in webpack config Oct 17, 2018
404.html Finalize cloud.gov migration and restore data links (#408) Nov 29, 2016
CONTRIBUTING.md remove the 18F-specific language Jan 14, 2015
Dockerfile code review Apr 21, 2017
Dockerfile.production Update production Docker image to use custom nginx config May 3, 2017
Gemfile Add versions to gems in Gemfile Jun 27, 2017
Gemfile.lock Update jekyll from 3.5.0 to 3.5.2 (#515) Sep 20, 2018
LICENSE.md list exceptions to license, linking to their licenses Jan 26, 2015
Makefile Update production Docker image to use custom nginx config May 3, 2017
README.md add https to readme Nov 14, 2018
Staticfile Deploying to cloud gov (#404) Nov 17, 2016
_config.yml Removing shutdown banner Jan 23, 2018
_development.yml Finalize cloud.gov migration and restore data links (#408) Nov 29, 2016
_staging.yml Finalize cloud.gov migration and restore data links (#408) Nov 29, 2016
data.html move variables into includes so they don't have to be declared in new… Jun 3, 2016
developer.md updating sample link after api key generated Jun 20, 2017
docker-compose.yml fix make command Apr 21, 2017
index.html move charts to include and refactor layouts to use that May 26, 2016
manifest.yml Finalize cloud.gov migration and restore data links (#408) Nov 29, 2016
nginx.conf Redirect /developers to /developer (#476) May 30, 2017
package-lock.json add formatter tests Nov 7, 2018
package.json add formatter tests Nov 7, 2018
system-security-plan.yml changing link Apr 19, 2017
webpack.config.js add webpack npm scripts Oct 25, 2018

README.md

Code Climate CircleCI Dependency Status

analytics.usa.gov

A project to publish website analytics for the US federal government.

For a detailed description of how the site works, read 18F's blog post on analytics.usa.gov.

Other organizations who have reused this project for their analytics dashboard:

The City of Anchorage, AK The Town of Apex, NC
The City of Boulder, CO The City of Chesapeake, VA
The City of Concord, NC The City of Eagle Mountain, UT
The City of Evanston, IL The City of Los Angeles, CA
The City of New Orleans, LA The City of Newark, NJ
The Borough of Norristown, PA The City of Omaha, NE
The City of Philadelphia, PA The City of Pleasanton, CA
The City of Princeton, NJ The City of Sacramento, CA
The City of San Francisco, CA The City of San Leandro, CA
The City of Santa Monica, CA Carbarrus County, NC
Cook County, IL data.jerseycitynj.gov
data.seattle.gov Douglas County, NE
Moulton Niguel Water District NYSERDA
Washington State University Rowan County, NC
The States of Jersey Tennessee Dept of Environment and Conservation
U.S. Department of Education U.S. Department of Veterans Affairs

This blog post details their implementations and lessons learned.

About the components

Ths app uses Jekyll to build the site, and Sass, Bourbon, and Neat for CSS.

The javascript provided is a webpacked aggregation of several different modules, leveraging d3 for the visualizations. Learn more on the webpack configuration

Developing locally

There are a couple of different ways to develop locally. Either using docker or running without docker.

Setup using Docker

You need Docker and docker-compose.

To build and run the app with docker-compose, run docker-compose up -d then you can access the app from http://localhost:4000, as the local filesystem is mounted on top of the docker container you can edit the files as you are developing locally.

  • this does not yet run the webpack script.

To see the jekyll logs, run:

docker-compose logs -f

Running locally without docker.

Run Jekyll with development settings:

make dev
npm install
npm run build-dev

(This runs bundle exec jekyll serve --watch --config=_config.yml,_development.yml.)

Adding Additional Agencies

  1. Ensure that data is being collected for a specific agency's Google Analytics ID. Visit 18F's analytics-reporter for more information. Save the url path for the data collection path.
  2. Create a new html file in the _agencies directory. The name of the file will be the url path.
touch _agencies/agencyx.html
  1. Create a new html file in the _data_pages directory. Use the same name you used in step 2. This will be the data download page for this agency
touch _data_pages/agencyx.html
  1. Set the required data for for the new files. (Both files need this data.) example:
---
name: Agency X # Name of the page
slug: agencyx # Same as the name of the html files. Used to generate data page links.
layout: default # type of layout used. available layouts are in `_layouts`
---
  1. Agency page: Below the data you just entered, include the page content you want. The _agencies page will use the charts.html partial and the _data_pages pages will use the data_download.html partial. example:
{% include charts.html %}

Developing with local data

The development settings assume data is available at /fakedata. You can change this in _development.yml.

Developing with real live data from analytics-reporter

If also working off of local data, e.g. using analytics-reporter, you will need to make the data available over HTTP and through CORS.

Various tools can do this. This project recommends using the Node module serve:

npm install -g serve

Generate data to a directory:

analytics --output [dir]

Then run serve from the output directory:

serve --cors

The data will be available at http://localhost:3000 over CORS, with no path prefix. For example, device data will be at http://localhost:3000/devices.json.

Javascript Modules

  • Index - includes the main dom selection and rendering queue of components, and the entry point for the webpack bundler.
  • lib/barchart the d3 configuration of the bar charts
  • lib/blocks an object of the specific components
  • lib/consoleprint the console messages displayed to users
  • lib/exceptions agency data to be changed by discrete exception rules
  • lib/formatters methods to help format the display of visualization scales and values
  • lib/renderblock d3 manipulator to load and render data for a component block
  • lib/timeseries the d3 configuration of the timeseries charts
  • lib/transformers helper methods to manipulate and consolidate raw data into proportional data.

Deploying the app

To deploy to analytics.usa.gov after building the site with the details in _config.yml:

make deploy_production

To deploy to analytics-staging.app.cloud.gov after building the site with the details in _config.yml and _staging.yml:

make deploy_staging

Deploying the app using Docker

NOTE: 18F does not use Docker in production!

If you are using Docker in production and you want to deploy just the static pages, you can build an nginx container with the static files built in, running the following command:

make docker-build-production PROD_IMAGE=yourvendor/your-image-name PROD_TAG=production

The resulting image will be an nginx server image that you can safely push and deploy to your server.

The image accepts an environment variable to specify the S3 URL that data at /data/* is served from:

docker run -p 8080:80 -e S3_BUCKET_URL=https://s3-us-gov-west-1.amazonaws.com/your-s3-bucket/data yourvendor/your-image-name:production

Building & Pushing Docker Images

This repo has git tags. The tag for Docker images built for this repo relate to these git tags. In the examples below, <version refers to the tag value of the current commit. When building a new version, be sure to increment the git tag appropriately.

When building images there are 2 images to build: <version> and <version>-production.

To build the images:

docker build -f ./Dockerfile -t 18fgsa/analytics.usa.gov:<version> .
docker build -f ./Dockerfile.production -t 18fgsa/analytics.usa.gov:<version>-production .

To push the images:

docker push 18fgsa/analytics.usa.gov:<version>
docker push 18fgsa/analytics.usa.gov:<version>-production

Environments

Environment Branch URL
Production master https://analytics.usa.gov
Staging master https://analytics-staging.app.cloud.gov

Webpack Configuration

The application compiles es6 modules into web friendly js via Wepback and the babel loader.

The webpack configuration is set in the wepback.config.js.

The current configuration uses babel present-env.

The webpack also includes linting using eslint leveraging the AirBnb linting preset.

The webconfig uses the UglifyJSPlugin to minimize the bundle.

The resulting uglified bundle is build into assest/bundle.js.

NPM webpack commands

Command purpose
npm run build-dev a watch command rebuilding the webpack with a development configuration (i.e. no minifiecation)
npm run build-prod a webpack command to build a minified and transpiled bundle.js

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.