In regression, you are predicting values so it made sense to discuss error as a distance of how far off our estimates were. In classifying a binary variable however, a model is either correct or incorrect. As a result, we tend to deconstruct this as how many false positives versus false negatives we come across.
In particular, we examine a few different specific measurements when evaluating the performance of a classification algorithm. In this review lab, we'll review precision, recall, accuracy, and F1-score in order to evaluate our logistic regression models.
You will be able to:
- Understand and assess precision, recall, and accuracy of classifiers
- Evaluate classification models using various metrics
Let's take a moment and review some classification evaluation metrics:
At times, it may be superior to tune a classification algorithm to optimize against precision or recall rather than overall accuracy. For example, imagine the scenario of predicting whether or not a patient is at risk for cancer and should be brought in for additional testing. In cases such as this, we often may want to cast a slightly wider net, and it is much preferable to optimize for recall, the number of cancer positive cases, then it is to optimize precision, the percentage of our predicted cancer-risk patients who are indeed positive.
import pandas as pd
df = pd.read_csv('heart.csv')
#Your code here
#Your code here
def precision(y_hat, y):
#Your code here
def recall(y_hat, y):
#Your code here
def accuracy(y_hat, y):
#Your code here
def f1_score(y_hat,y):
#Your code here
Do this for both the training and the test set
#Your code here
Great Job! Now it's time to check your work with sklearn.
Each of the metrics we calculated above is also available inside the sklearn.metrics
module.
In the cell below, import the following functions:
precision_score
recall_score
accuracy_score
f1_score
Compare the results of your performance metrics functions with the sklearn functions above. Calculate these values for both your train and test set.
#Your code here
Calculate and then plot the precision, recall, accuracy, and F1-score for the test and train splits using different train set sizes. What do you notice?
importimport matplotlib.pyplotmatplot as plt
%matplotlib inline
training_Precision = []
testing_Precision = []
training_Recall = []
testing_Recall = []
training_Accuracy = []
testing_Accuracy = []
for i in range(10,95):
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= None) #replace the "None" here
logreg = LogisticRegression(fit_intercept = False, C = 1e12)
model_log = None
y_hat_test = None
y_hat_train = None
# 6 lines of code here
Create 4 scatter plots looking at the test and train precision in the first one, test and train recall in the second one, testing and training accuracy in the third one, and testing and training f1-score in the fourth one.
# code for test and train precision
# code for test and train recall
# code for test and train accuracy
# code for test and train F1-score
Nice! In this lab, you gained some extra practice with evaluation metrics for classification algorithms. You also got some further python practice by manually coding these functions yourself, giving you a deeper understanding of how they work. Going forward, continue to think about scenarios in which you might prefer to optimize one of these metrics over another.