Metro2 Evaluators

Evaluators are logic checks that find inconsistencies in Metro2 data. The Metro2 evaluator tool runs this set of checks on every record in a Metro2 dataset and saves the results. Then, users can use the Metro2 UI to view and analyze the results.

Contents:

• Creating evaluators
• Handling evaluator metadata
• How to run the evaluators

Creating evaluators

Developers may define evaluators by adding them in code. To add an evaluator, three changes must be made (in any order):

Step 1: Add the evaluator to the list of Evaluator Metadata in the Metro2 database. This allows the Metro2 application to display relevant information about the evaluator in the UI to support users' analysis of the data. Do this by adding a row to eval_metadata.csv with information about the evaluator. See handling evaluator metadata below for more information about evaluator metadata and the CSV format.

Step 2: Add the evaluator definition in code. An evaluator is a Python method that takes one parameter: a Django QuerySet of AccountActivity objects, which is the whole set of records in the Metro2 dataset. The method should return a filtered subset of the queryset using filtering methods (e.g. filter and exclude). We also recommend adding a Python test to confirm that your evaluator behaves as desired. For an example of evaluator code, see the definition and test for Portfolio-Type-1.

Step 3: Include the evaluator in the Django configuration list of METRO2_EVALUATORS. This list is a Python dictionary where each key is an evaluator ID, whose value is the Python path where the evaluator definition can be found. This associates the evaluator ID with its definition and allows the evaluator process to find and run the evaluator.

Handling evaluator metadata

Each evaluator has several metadata fields associated with it, such as name, short description, long description, fields used, rationale, and more. We seed the database with initial metadata about each evaluator, then allow users to modify some of the fields.

Evaluator CSV format

When importing and exporting evaluator metadata, we use a CSV with the following columns: id,category,description,long_description,fields_used,fields_display,rationale,potential_harm,alternate_explanation,crrg_reference

• id: this column is what we use to connect the evaluator metadata to the evaluator function, which is defined in code. This means that the id column needs to exactly match the name of the function in the code. If the names don't match, any evaluator results won't be correctly associated with the evaluator metadata in the system.

• category: A short phrase that can be used to group related evaluators. The results table on the event overview page displays and can be sorted by category.

• description: A brief, high-level explanation of what the evaluator does. This description is used throughout the tool – on the event overview, evaluator, and account pages -- to provide basic context about the evaluator.

• long_description: A longer, pseudo-code description of the specific fields the evaluator looks at and the values it is looking for in those fields. This is displayed on the evaluator page to help users understand and assess the evaluator results.

  Long descriptions can be complicated and benefit from formatting, but the tool does not currently provide a way to store formatted data for this field. It expects to import the field's content -- unformatted except for new lines -- from a csv, and doesn't provide an interface to add and save formatting to it. As a workaround, the front end applies some basic formatting using the following logic: one new line character indicates a new paragraph, two new line characters indicate a new section, and the first line of a section is formatted as a header if it doesn't contain pseudocode symbols (=, ≠, <, >, etc). An example of how long description content created in Excel would be output in the tool:

  Excel	CSV	Tool

• fields_used: A list of fields used by the evaluator to perform its logic check. The results table on the evaluator page will display the fields in this list and the evaluator’s fields_display list, as well as some constants defined in the django code.

• fields_display: A list of any additional evaluator-specific fields that should be displayed in the results table for this evaluator.

• rationale: An explanation of why the inconsistency found by this evaluator is a problem and what’s required for the account to be accurately furnished. Displayed in the How to evaluate these results section on the evaluator page.

• potential_harm: The negative impact this inconsistency might have on an individual’s credit. Displayed in the How to evaluate these results section on the evaluator page.

• alternate_explanation: Any logical reasons for this inconsistency to exist when the account has been accurately furnished. Displayed in the How to evaluate these results section on the evaluator page.

• crrg_reference: References to specific sections of the Credit Reporting Resource Guide (CRRG) that may be useful for interpreting the results of this evaluator. Displayed in the How to evaluate these results section on the evaluator page.

Importing metadata

Do this when deploying the project to a new environment to create evaluator metadata records in the database.

How to import the evaluator metadata into the system:

1. Create a CSV of all known evaluator metadata using the format described above.
2. Save the CSV to this repo using the following filename: cfpb_evaluators/eval_metadata.csv.
3. Import the metadata by running the following Django management command in the environment where the metadata should be imported: python manage.py import_evaluator_metadata.
   • This command will update any existing records with the new metadata and create any that don't already exist. It won't delete existing records that are missing from the csv.

Exporting metadata

Do this when users have made manual updates to the evaluator metadata and you want to propagate those updates to another environment.

How to export the evaluator metadata:

1. Visit the /api/all-evaluator-metadata endpoint for the environment in the browser.
   • This will download a CSV of all evaluator metadata in the system, which you can import into any Metro2 environment.

How to run evaluators

After a dataset has been parsed and is saved in the M2 database, the evaluators can be used to analyze and find inconsistencies in the data. To do so, use the evaluate management command: python manage.py evaluate -e [event_id]. This will run all of the evaluators that have been configured in the METRO2_EVALUATORS Django setting.

When the evaluators are run, the results are saved in the M2 database. When the process is finished, the results are available to be viewed in the Metro 2 application.