Skip to content

Conversation

@s-good
Copy link
Contributor

@s-good s-good commented Aug 16, 2017

This adds some code I wrote a while ago to generate a ROC curve (graph of true positive rate versus false positive rate) by optimally combining the results of different tests together to give the best possible TPR for any FPR. I've adapted it to the new sqlite database infrastructure and I've moved some of the database routines into a new dbutils.py file as they are used by both summarize-results and the new analyse-results programs.

The analyse-results program is fairly simple. It does an AND on the different QC tests to make combinations and calculates the TPR and FPR of each. The test combination that has the highest ratio of TPR increase to FPR increase is selected to make the next point in the ROC curve, and this continues until there are no more test combinations that give a useful increase in TPR. The program seems to work fairly well given the simple approach, but memory requirements will start to become an issue if the number of QC checks expand much.

I've also included functionality that checks the CSIRO_wire_break results and only uses profile levels above an XBT wire break, in order to allow us to concentrate on other error modes.

@s-good s-good requested a review from bkatiemills August 16, 2017 13:18
@bkatiemills
Copy link
Member

Looks good to me on first inspection, thanks @s-good! We ought to write some unit tests for all these helper functions, and also check and see if there's other places in the code we ought to be using them - I'll open a couple of issues and look into them shortly.

@bkatiemills bkatiemills merged commit 2554710 into IQuOD:no-global Aug 20, 2017
@s-good s-good deleted the analyse-results branch May 8, 2018 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants