Skip to content

Testing and Graphing Results

Jump to bottom Edit New page
cpb8010 edited this page Sep 2, 2013 · 5 revisions

This page covers the functionality of the Python scripts in reporting.

crossValTest.py is an example file showing how to build, run, and graph the results of an MC-VOSM test run. To run MC-VOSM on another database, new functions will be necessary to setup the tests. The running and graphing portion of the Python scripts should not need to be modified.

Test setup

The bs_helper.py is specifically for the Bosphouros Database format. It handles projecting 3D faces to 2D images and registers those projections to a gray-scale face. The faces and annotation points are then scaled and saved based on the groups in the database. Filtering, sorting, converting, and then saving groups of files is necessary before other Python functions start performing tests. These setup steps could be done manually, but creating a Python function allows for the creation of multiple test folds and randomized train/test subjects. The example Python script saves the state of each of the intermediate steps to reduce duplication and allow for faster changes in tests.

Following the example

The following sections explain the setup steps as they are performed in crossValTest.py. Each of these steps performs a save after they successfully complete, to re-run a step, delete the save file in the test root directory.

Filtering

Given the database path and a list of required landmarks, return only the faces that have all of the required landmarks. This step requires reading all of the annotation files (storing file name and contents) and navigating the database directory. Saved to subjectList.pkl

Sorting

With a list of filtered faces, group based on file name. This step does not read from the database directory. Saved to sortedFaces.pkl

Convert images

Load and convert the images from the database, keeping the filtered group structure. This is the first step to require image processing and produces images paired with annotations. Saved to convertedFaces.pkl

Write images and annotations to disk

This step is performed during each cross validation fold; previous steps to this will not change depending on the train/test subject split. This step requires a list of training and test subjects. The images and annotation files are then written to named group folders, where the training and test groups can be different. Saved to Train and Test folders.

Final File/Directory Structure

If the number of cross-validation folds is non-zero, the test root directory should have folders named Set_0 through Set_N. Each Set directory should contain a directory called Faces. Each Faces directory will contain two folders named Train and Test. The Train directory will have one folder per training model; the Test directory will have one folder per testing subset. The train and test folders will contain both face images and annotations.

Building Models & Running tests

Each validation fold is an independent train / test run. If 0 validation folds are specified, all runs will be performed in the test root directory. These functions are performed by vosm_helper.py, as they are not specific to any database.

Building

Requires the path to a working test_smbuilding program. This step will produce results in a Models folder within each Set directory. The number and content of the models is determined by the same groups that created the folders in the Train directory. The arguments to the building program are static, as a single multichannel model can be used to test many combinations of channels. The building process will report if some models did not build, but will continue. If the building process is interrupted for any reason, it is suggested to delete the Models directory and start again.

Testing

Requires the path to a working test_smfitting program. This step will save the fitting results in a Results folder within each Set directory. The test groups are determined by the same groups created the folders in the Test directory. Variable lists of arguments are used to test the many combinations of fitting techniques, channel selections and initial shape placements. All of these combinations are tested exhaustively, so this step can take a significant amount of time (days or more) depending on the number of variable arguments, test groups, training models, and images. The fitting process will report if models could not be found, images could not be found, or if the fitting program fails for another reason. If the fitting step is interrupted before completion, it must be restarted to prevent duplicate results. Checking if a test has already ran would be a good enhancement, but this should rarely occur

Collecting results

Fitting results are stored in text files across multiple directories, to collect large quantities of results vosm_helper also contains result scraping functions. These are not specific to any database. The results are loaded into an SQLite database by the Python functions in vosm_helper.py. Each validation fold contains it's own database file.

Database format

The format is also in the comments of the CreateDB function. There are 7 tables:

  • SummaryResults table: 11 columns(5 inputs, 6 outputs), 1 row per test run
    • First 5 Columns, Inputs
      • verfSet INTEGER, locTech TEXT, trainModel TEXT, fitTech TEXT, testSet TEXT
    • Next 6 Columns, Outputs
      • detTimes INTEGER, avgIterations REAL, avgFitTime REAL, avgDist REAL, avgDev REAL, stdDev REAL
  • 2 Individual result tables: 1 final result table, 1 intermediate result table
    • PtsErrors table: 1+numPts columns(1 primary key, n points), 1 row per fitting iteration+1
    • Pts table: 1 + 2*numPts columns(1 primary key, 2n columns), 1 rows per fitting iteration+1
  • 4 Individual result sub-tables: 2 error tables, 2 point storage tables
    • CPtsErrors: 4 columns(1 primary key, 3 outputs), 1 row per fitting iteration+1
      • key INTEGER PRIMARY KEY,leftEye REAL,rightEye REAL,mouth REAL
    • CPts: 5 columns(1 primary key, 1 input, 3 outputs), 2 rows per fitting iteration+1
      • key INTEGER PRIMARY KEY, leftEyeX REAL, leftEyeY REAL, rightEyeX REAL, rightEyeY REAL, mouthX REAL, mouthY REAL
    • FinalIndvResults table: 11 columns(6 inputs, 4 linked tables, 1 output), 1 row per image
      • verfSet INTEGER, subNum INTEGER, locTech TEXT, trainModel TEXT, fitTech TEXT, testSet TEXT, errPtsTableKey INTEGER, ptsTableKey INTEGER, errCpTableKey INTEGER, cpTableKey INTEGER, fitTime REAL, FOREIGN KEY(errPtsTableKey) REFERENCES PtsErrors(key), FOREIGN KEY(ptsTableKey) REFERENCES Pts(key) , FOREIGN KEY(errCpTableKey) REFERENCES CPtsErrors(key), FOREIGN KEY(cpTableKey) REFERENCES CPts(key)
    • InterIndvResults table: 9 columns(7 inputs, 2 linked tables), 1 row per fitting iteration
      • verfSet INTEGER, subNum INTEGER, locTech TEXT, trainModel TEXT, fitTech TEXT, testSet TEXT, iterNum INTEGER, errPtsTableKey INTEGER, ptsTableKey INTEGER, FOREIGN KEY(errPtsTableKey) REFERENCES PtsErrors(key), FOREIGN KEY(ptsTableKey) REFERENCES Pts(key)

Querying and Graphing Results

Queries over the individual SQLite databases returns only results for that validation fold, the graphing functions get the results from all the databases before creating graphs. The graphs functions take parameters to display different slices of data with either bar charts or line graphs, depending on the type of data requested. This automatic graphing technique can produce 1000's of graphs using the many parameters available. [crossValTest.py](../blob/master/reporting/crossValTest.py shows how to call the graphing functions.

Add a custom footer
Add a custom sidebar
Clone this wiki locally