Monitoring Facebook Political Ads
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

Facebook Political Ad Collector

This is the source code behind our project to collect political ads on Facebook. Built versions are available for Firefox and Chrome. You can browse the American ads we've collected at ProPublica, and the Australian ads over on the Guardian's website.

We're asking our readers to use this extension when they are browsing Facebook. While they are on Facebook a background script runs to collect ads they see. The extension shows those ads to users and asks them to decide whether or not a particular ad is political. Serverside, we use those ratings to train a naive bayes classifier that then automatically rates the other ads we've collected. The extension also asks the server for the most recent ads that the classifier thinks are political so that users can see political ads they haven't seen. We're careful to protect our user's privacy by not sending identifying information to our backend server.

We're open sourcing this project because we'd love your help. Collecting these ads is challenging, and the more eyes on the problem the better.

Download and Try It

Build and Develop Locally

Extension

The extension popup is a preact application and you can build a development version by running the following:

$ cd extension
$ npm install
$ npm run watch

If you are a Firefox user you can open a clean browser instance with:

$ npm run ff

and any changes will automatically refresh the extension. (You'll need webpack installed globally.)

In Chrome you'll need to add an unpacked extension by following these directions.

Backend

First, make sure you have Rust installed:

$ curl -sSf https://static.rust-lang.org/rustup.sh | sh

Be sure to add ~/.cargo/bin (or wherever cargo is installed, path/to/.cargo/bin) to your PATH. You can do this by adding this line to your .bash_rc or .zshrc or whatever config file you typically use for your shell.

PATH=$PATH:~/.cargo/bin

The backend server is a rust application that runs on top of diesel and hyper. You'll need the diesel command line interface to get started and to create the database:

$ cargo install diesel_cli
$ diesel database setup

You'll also need to clone the fbpac-api app and run its migrations, to add a few other columns. In a separate directory:

$ git clone https://github.com/propublica/fbpac-api-public.git
$ cd fbpac-api-public
$ bundle install
$ rake db:migrate

You can kick the tires by running:

$ cd backend/server
$ cargo build
$ cargo run

This will give a server running at localhost:8080. You will also need to build the backend's static resources. To do this, in another terminal tab:

$ cd backend/server/public
$ npm install
$ NODE_ENV=development npm run watch

This will build the required static assets (javascript & css) to view the admin console at localhost:8080/facebook-ads/.

If you make any changes to the database, after you run the migration, you'll want to update the schema with diesel print-schema > src/schema.rs. You'll need to do this even if you make changes via a Rails migration.

The backend has both unit and integration tests. You will need to set up a test database alongside your actual database in order to run the integration tests. To do this, you will need to the same as above, but substitute out the database test URL:

$ diesel database setup --database-url postgres://localhost/facebook_ads_test

Note that the value for --database-url came from the TEST_DATABASE_URL value set in the .env file. Make sure that the urls match before you run the tests!

Additionally, because the integration tests use the database, we want to make sure that they aren't run in parallel, so to run the tests:

$ RUST_TEST_THREADS=1 cargo test

This will run the tests in sequence, avoiding parallelism errors for tests that require the database.

Classifier

We train the classifier using python and scikit learn and the source is in backend/classifier/. We're using pipenv to track dependencies.

To download pipenv:

$ brew install pipenv

To get started you can run:

$ cd backend/classifier/
$ pipenv install
$ pipenv shell

To download the seeds for the classifier, you'll need a Facebook app with the proper permissions and you'll run the seed command like this:

$ FACEBOOK_APP_ID=whatever FACEBOOK_APP_SECRET=whatever DATABASE_URL=postgres://whatever/facebook_ads ./classify seed en-US`

Alternatively, you can build the model without seeds, relying instead just on votes in the extension and suppressions in the admin. And to build the classifier you'll want to run:

$ pipenv run ./classify build

To classify the ads you've collected you can run:

$ pipenv run ./classify classify

You can download pre-trained models with pipenv run ./classify get_models.

Internationalization and Localization

Translations for the extension are stored in extension/_locales/${locale}/messages.json.

A locale is a ISO 639-1 language code (e.g. en, de) with an optional ISO 3166-1 Alpha-2 country suffix (e.g. de_CH).

Users can select from all known languages and countries while onboarding. The UI then uses the first available translation in following order: ${langauge}_${country}, ${langauge}, en.

Active Languages and Countries

In extension/src/i18n.js a list of active language and country codes can be defined. Active ones get prioritised in the UI.

Locale Specific Styles in Popup

You can customize, for example font sizes, with [lang] and [data-locale] CSS selectors:

[lang=de] .toggle {
  font-size: 0.78rem;
}
[data-locale=de_CH] .toggle {
  font-size: 0.78rem;
}

Stories

Where We Need Your Help

In general, the project needs more tests. We've written a couple of tests for parsing the Facebook timeline in the extension directory, and a few for the tricky bits in the server, but any help here would be great!

Also, the rust backend needs a bit of love and care, and there is a bit of a mess in backend/server/src/server.rs that could use cleaning up.