New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#271 LUNA16 evaluation #292

Merged
merged 2 commits into from Jan 22, 2018

Conversation

Projects
None yet
3 participants
@vessemer
Copy link
Contributor

vessemer commented Jan 22, 2018

This allows to compute FROC and CPM metrics with bootstrapping for the classification and detection tasks.

Description

Free-Response Receiver Operating Characteristic (FROC) and competition performance metric (CPM). It computes an average of the seven sensitivities measured at several false positives per scan (FPPS) thresholds, more concretely, at each FPPS ∈ {0.125, 0.25, 0.5, 1, 2, 4, 8} true positive rate was computed. Mean of which forms the CPM as discussed in #271.

Reference to official issue

This reference to the #271.

How Has This Been Tested?

froc_3dlrcnn

CPM over 10-Fold cross validation:

0.125 0.25 0.5 1 2 4 8 Score (CPM)
0.595 0.670 0.731 0.793 0.835 0.868 0.887 0.76

CLA

  • I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
@WGierke

This comment has been minimized.

Copy link
Contributor

WGierke commented Jan 22, 2018

Nice! You mentioned that you used 10-fold cross validation. Can you document somewhere how you trained and evaluated the model using your newly introduced methods?

@vessemer

This comment has been minimized.

Copy link
Contributor

vessemer commented Jan 22, 2018

I've used my old pipeline since it fits my currently available one TITAN X GPU, though the pipeline code in my repo looks not that good :)

@vessemer

This comment has been minimized.

Copy link
Contributor

vessemer commented Jan 22, 2018

Three months ago I've worked on algorithms unification and training parts, but then master branch diverges from mine and I abandoned that idea for a while. The code I've written lies here. I hope to return to this soon.

@vessemer

This comment has been minimized.

Copy link
Contributor

vessemer commented Jan 22, 2018

Btw, for 10 fold cross-validation was used split from the data page suggested by the LUNA16 authors.

@lamby

This comment has been minimized.

Copy link
Contributor

lamby commented Jan 22, 2018

This is cool!

@lamby lamby merged commit b924809 into drivendataorg:master Jan 22, 2018

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
concept-to-clinic/cla @vessemer has signed the CLA.
Details

@vessemer vessemer deleted the vessemer:271_LUNA_evaluation branch Jan 22, 2018

This was referenced Jan 23, 2018

@vessemer vessemer referenced this pull request Feb 1, 2018

Merged

Reactivate tests and clear models #306

1 of 1 task complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment