-
Notifications
You must be signed in to change notification settings - Fork 3
Metro2 Evaluators
Evaluators are logic checks that find inconsistencies in Metro2 data. The Metro2 evaluator tool runs this set of checks on every record in a Metro2 dataset and saves the results. Then, users can use the Metro2 UI to view and analyze the results.
Contents:
Developers may define evaluators by adding them in code. To add an evaluator, three changes must be made (in any order):
Step 1: Add the evaluator to the list of Evaluator Metadata in the Metro2 database.
This allows the Metro2 application to display relevant information about the evaluator in the UI to support users' analysis of the data.
Do this by adding a row to eval_metadata.csv with information about the evaluator.
See handling evaluator metadata below for more information about evaluator metadata and the CSV format.
Step 2: Add the evaluator definition in code.
An evaluator is a Python method that takes one parameter: a Django QuerySet of AccountActivity objects, which is the whole set of records in the Metro2 dataset.
The method should return a filtered subset of the queryset using filtering methods (e.g. filter and exclude).
We also recommend adding a Python test to confirm that your evaluator behaves as desired.
For an example of evaluator code, see the definition and test for Portfolio-Type-1.
Step 3: Include the evaluator in the Django configuration list of METRO2_EVALUATORS.
This list is a Python dictionary where each key is an evaluator ID, whose value is the Python path where the evaluator definition can be found.
This associates the evaluator ID with its definition and allows the evaluator process to find and run the evaluator.
Each evaluator has several metadata fields associated with it, such as name, short description, long description, fields used, rationale, and more. We seed the database with initial metadata about each evaluator, then allow users to modify some of the fields.
When importing and exporting evaluator metadata, we use a CSV with the following columns:
id,category,description,long_description,fields_used,fields_display,rationale,potential_harm,alternate_explanation,crrg_reference
The id column is what we use to connect the evaluator metadata to the evaluator function, which is defined in code.
This means that the id column needs to exactly match the name of the function in the code.
If the names don't match, any evaluator results won't be correctly associated with the evaluator metadata in the system.
Do this when deploying the project to a new environment to create evaluator metadata records in the database.
How to import the evaluator metadata into the system:
- Create a CSV of all known evaluator metadata using the format described above.
- Save the CSV to this repo using the following filename:
cfpb_evaluators/eval_metadata.csv. - Import the metadata by running the following Django management command in the environment where the metadata should be imported:
python manage.py import_evaluator_metadata.- This command will update any existing records with the new metadata and create any that don't already exist. It won't delete existing records that are missing from the csv.
Do this when users have made manual updates to the evaluator metadata and you want to propagate those updates to another environment.
How to export the evaluator metadata:
- Visit the
/api/all-evaluator-metadataendpoint for the environment in the browser.- This will download a CSV of all evaluator metadata in the system, which you can import into any Metro2 environment.
After a dataset has been parsed and is saved in the M2 database, the evaluators can be used to analyze and find inconsistencies in the data.
To do so, use the evaluate management command: python manage.py evaluate -e [event_id].
This will run all of the evaluators that have been configured in the METRO2_EVALUATORS Django setting.
When the evaluators are run, the results are saved in the M2 database. When the process is finished, the results are available to be viewed in the Metro 2 application.