Skip to content
General Purpose Risk Modeling and Prediction Toolkit for Policy and Social Good Problems
Branch: master
Clone or download
Latest commit 18368c1 Jun 5, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Scheduled monthly dependency update for May (#679) May 30, 2019
example/config Bias part 2 (#688) May 30, 2019
requirement Scheduled monthly dependency update for May (#679) May 30, 2019
src Added DummyClassifier to the SimpleClassifiers batch (#702) May 30, 2019
.bandit.yml Make codacy stop complaining about asserts (#383) Feb 8, 2018
.codeclimate.yml Remove duplicate/similar checks from codeclimate and bump line length… Oct 11, 2018
.editorconfig Project skeleton populated by cookiecutter Oct 17, 2016
.gitignore Add docs/site to gitignore May 6, 2019
.python-version.current development environment wizard (#511) Nov 28, 2018
.pyup.yml Switch back to monthly now that we are pinned Sep 13, 2018
.travis.yml Support Python 3.7 [Resolves #683] (#684) May 6, 2019
AUTHORS.rst completed integration of results-schema as triage component Dec 12, 2017
CONTRIBUTING.md Fix Travis deploy [Resolves #493] (#677) Apr 25, 2019
HISTORY.rst Cookie Cutter defaults Oct 19, 2016
LICENSE MS Triage (#666) Apr 9, 2019
MANIFEST.in completed integration of results-schema as triage component Dec 12, 2017
README.rst Update README.rst Jun 5, 2019
develop support in `develop` script for detection of pyenv installed via Home… Dec 11, 2018
manage.py Fix Travis deploy [Resolves #493] (#677) Apr 25, 2019
pytest.ini updated build/test configuration to reflect moved source files Nov 15, 2017
setup.cfg Bump version: 3.2.1 → 3.3.0 Feb 20, 2019
setup.py Support Python 3.7 [Resolves #683] (#684) May 6, 2019
tox.ini Support Python 3.7 [Resolves #683] (#684) May 6, 2019

README.rst

Triage

Risk modeling and prediction

https://travis-ci.org/dssg/triage.svg?branch=master https://codeclimate.com/github/dssg/triage.png

Building systems that use predictive models require answering many design decisions, turning them into modeling choices, and technical tasks. Questions such as cohort selection, unit of analysis determination, outcome determination, feature (explanantory variables) generation, model/classifier training, evaluation, selection, and list generation are often complicated and hard to choose apriori. In addition, once these choices are made, they have to be combined in different ways throughout the course of a project.

Triage aims to help solve these problems by:

  1. Guiding users (data scientists, analysts, researchers) through these design choices by highlighting operational use questions that are important.
  2. Providing interfaces to these different phases of a project, such as an Experiment. Each phase is defined by a configuration (corresponding to a design choice) specific to the needs of the project, and an arrangement of core data science components that work together to produce the output of that phase.

Installation

Prerequisites

To use Triage, you first need:

  • Python 3+
  • A PostgreSQL database with your source data (events, geographical data, etc) loaded.
  • Ample space on an available disk, (or for example in Amazon Web Services's S3), to store the needed matrices and models for your experiment

Building

Triage is a Python package distributable via setuptools. It may be installed directly using easy_install or pip, or named as a dependency of another package as triage.

To build this package (without installation), its dependencies may alternatively be installed from the terminal using pip:

pip install -r requirement/main.txt

Testing

To add test (and development) dependencies, use test.txt:

pip install -r requirement/test.txt [-r requirement/dev.txt]

Then, to run tests:

pytest

Development

To quickly bootstrap a development environment, having cloned the repository, invoke the executable develop script from your system shell:

./develop

A "wizard" will suggest set-up steps and optionally execute these, for example:

(install) begin

(pyenv) installed ✓

(python-3.6.2) installed ✓

(virtualenv) installed ✓

(activation) installed ✓

(libs) install?
1) yes, install {pip install -r requirement/main.txt -r requirement/test.txt -r requirement/dev.txt}
2) no, ignore
#? 1

Experiment

The first phase implemented in Triage is the Experiment. An experiment represents the initial research work of creating design matrices from source data, and training/testing/evaluating a model grid on those matrices. At the end of the experiment, a relational database with results metadata is populated, allowing for evaluation by the researcher.

Documentation

Background

Triage is developed at the University of Chicago's Center For Data Science and Public Policy. We created it in response to commonly occuring challenges we've encountered and patterns we've developed while working on projects for our partners.

Major Components Used by Triage

Triage makes use of many core data science components developed at DSaPP. These components can be useful in their own right, and are worth checking out if you'd like to make use of a subset of Triage's functionality in an existing pipeline.

Components Within Triage

  • Architect: Plan, design and build train and test matrices. Includes feature and label generation.
  • Catwalk: Training, testing, and evaluating machine learning classifier models
  • Collate: Aggregation SQL Query Builder. This is used by the Architect to build features.
  • Timechop: Generate temporal cross-validation time windows for matrix creation
  • Results Schema: Generate a database schema suitable for storing the results of modeling runs

Design Goals

There are two overarching design goals for Triage:

  • All configuration necessary to run the full experiment from the external interface (ie, Experiment subclasses) from beginning to end must be easily serializable and machine-constructable, to allow the eventual development of tools for users to design experiments.
  • All core functionality must be usable outside of a specific pipeline context or workflow manager. There are many good workflow managers; everybody has their favorite, and core functionality should not be designed to work with specific execution expectations.

Future Plans

  • Generation and Management of lists (ie for inspections) by various criteria
  • Integration of components with various workflow managers, like Drain and Luigi.
  • Comprehensive leakage testing of an experiment's modeling run
  • Feature Generation Wizard
You can’t perform that action at this time.