Permalink
Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
73 lines (59 sloc) 3.63 KB

Note: the current releases of this toolbox are a beta release, to test working with Haskell's, Python's, and R's code repositories.

Metrics provides implementations of various supervised machine learning evaluation metrics in the following languages:

  • Python (easy_install ml_metrics)
  • R (install from source, hasn't been added to CRAN yet)
  • Haskell (cabal install Metrics)
  • MATLAB / Octave (clone the repo & run setup from the MATLAB command line)

For more detailed installation instructions, see the README for each implementation.

EVALUATION METRICS

Evaluation MetricPythonRHaskellMATLAB / Octave
Absolute Error (AE)
Average Precision at K (APK, AP@K)
Area Under the ROC (AUC)
Classification Error (CE)
Log Loss (LL)
Mean Log Loss (LogLoss)
Mean Absolute Error (MAE)
Mean Average Precision at K (MAPK, MAP@K)
Mean Quadratic Weighted Kappa
Mean Squared Error (MSE)
Mean Squared Log Error (MSLE)
Quadratic Weighted Kappa
Root Mean Squared Error (RMSE)
Root Mean Squared Log Error (RMSLE)
Squared Error (SE)
Squared Log Error (SLE)

TO IMPLEMENT

  • Gini
  • NormalizedGini
  • LevensteinDistance
  • F1 score
  • Multiclass log loss
  • Lift
  • Average Precision for binary classification
  • precision / recall break-even point
  • cross-entropy
  • True Pos / False Pos / True Neg / False Neg rates
  • precision / recall / sensitivity / specificity
  • mutual information

HIGHER LEVEL TRANSFORMATIONS TO HANDLE

  • GroupBy / Reduce
  • Weight individual samples or groups

PROPERTIES METRICS CAN HAVE

(Nonexhaustive and to be added in the future)

  • Min or Max (optimize through minimization or maximization)
  • Binary Classification
    • Scores predicted class labels
    • Scores predicted ranking (most likely to least likely for being in one class)
    • Scores predicted probabilities
  • Multiclass Classification
    • Scores predicted class labels
    • Scores predicted probabilities
  • Regression
  • Discrete Rater Comparison (confusion matrix)