Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Evaluation: Add more metrics for evaluating survival analysis tasks #9

Open
bcebere opened this issue Feb 6, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@bcebere
Copy link
Contributor

bcebere commented Feb 6, 2023

Feature Description

One of the major tasks of the library is evaluating the quality of the models and evaluating the AutoML objectives.

To that end, metrics are needed for every supported problem type.

One of them is evaluating survival analysis tasks. The library should offer an API for using any of these metrics, testing the predicted values against the ground truth.

The metrics should be reported by each evaluation time horizon, and aggregated(mean, std).

Important metrics to cover here:
[X] c_index : The concordance index or c-index is a metric to evaluate the predictions made by a survival algorithm. It is defined as the proportion of concordant pairs divided by the total number of possible evaluation pairs.
[X] brier_score: The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions.

  • aucroc : the Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
  • sensitivity: Sensitivity (true positive rate) is the probability of a positive test result, conditioned on the individual truly being positive.
  • specificity: Specificity (true negative rate) is the probability of a negative test result, conditioned on the individual truly being negative.
  • PPV: The positive predictive value(PPV) is the probability that following a positive test result, that individual will truly have that specific disease.
  • NPV: The negative predictive value(NPV) is the probability that following a negative test result, that individual will truly not have that specific disease.

AP reference: https://github.com/vanderschaarlab/autoprognosis/blob/main/src/autoprognosis/utils/tester.py

@bcebere bcebere added the enhancement New feature or request label Feb 6, 2023
@DrShushen DrShushen transferred this issue from another repository Mar 3, 2023
@DrShushen DrShushen changed the title [Evaluation] Add metrics for evaluating survival analysis tasks [Evaluation] Add more metrics for evaluating survival analysis tasks May 25, 2023
@DrShushen DrShushen changed the title [Evaluation] Add more metrics for evaluating survival analysis tasks [Enhancement] Evaluation: Add more metrics for evaluating survival analysis tasks Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant