Skip to content
Script to transform the Disconnect block-list into Safebrowsing v2 format for Firefox Tracking Protection
Python
Branch: master
Clone or download
Latest commit 6c61186 Oct 11, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore Refactor list creation to use external list parser Mar 12, 2019
CODE_OF_CONDUCT.md Add Mozilla Code of Conduct file Mar 27, 2019
LICENSE Initial commit Sep 23, 2014
README.md update mappings and README Jun 1, 2018
constants.py Move list types to constants Sep 16, 2019
disconnect_mapping.json fix apps.fbsbx.com mapping Oct 11, 2019
lists2safebrowsing.py Spelling Sep 24, 2019
publish2cloud.py
requirements.txt Use packaging to check valid version Sep 20, 2019
sample_shavar_list_creation.ini Set versioning config Sep 20, 2019

README.md

shavar-list-creation

This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

Requirements

  • python 2.x
  • (optional) virtualenv and/or virtualenvwrapper

Run

  1. (optional) Make a virtualenv for the project and activate it:

    virtualenv shavar-list-creation
    source shavar-list-creation/bin/activate
    
  2. Install required libraries:

    pip install -r requirements.txt
    
  3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

    cp sample_shavar_list_creation.ini shavar_list_creation.ini
    
  4. Run the lists2safebrowsing.py script:

    ./lists2safebrowsing.py
    

Usage

This is run by a Jenkins deployment job every 30 minutes that:

  1. Checks out this repository
  2. Checks out the shavar-list-creation-config repository
  3. Copies stage.ini or prod.ini to shavar_list_creation.ini
  4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar.
You can’t perform that action at this time.