Repository navigation
Ch 3 Classification
Classification involves assigning data to specific labels. For example, predicting if a set of molecules under given conditions will react with each other (labels would be they do react or they don't react). Another example would be predicting what secondary structure a segment of amino acids folds into (alpha helix, beta sheet, random coil). Classifiers can also be made for continuous data by setting a threshold - for example setting a threshold where a compound is considered soluble if it is more soluble than the threshold and then classifying compounds as soluble or not. The different types of classification can be broken down into:
- Binary - two possible classes
- Multiclass - more than two classes but only one class per data point
- Multilabel - data point can have more than one label
- Hierarchical - there are subclasses within classes AI models can be used to predict labels for all of these cases.
The models need to predict the class label from some information about the data. The chosen information is commonly referred to as features and the process of extracting these features from the data is called featurization. The choice of features plays a large role in the downstream accuracy of models trained on the features because only features that are predictive of the label will contribute to an accurate model. In future lessons we will talk about strategies for analyzing your data to choose the best features. Expert knowledge is also important in choosing features. Chemists and biologists will have an intuition for what features will be most predictive in various chemistry of biology problems. Some common featurization strategies are:
- Molecular descriptors (e.g. number of H-bond donor/acceptors, molecular weight, etc.)
- Molecular fingerprints
- One hot encoding (especially for nucleic acid or protein sequences)
One hot encoding involves representing a sequence as a matrix of size NxL where N is the number of possible characters in the sequence and L is the length of the sequence. For example for a protein sequence of only canonical amino acids N=20. A 1 will be present if that position in the sequence is the corresponding character, and 0s will be used to fill the rest of the matrix. You could also simplify this representation - for example classify amino acids by their properties (hydrophobic, hydrophilic, positive charge, negative charge, aromatic) and then one hot encode on these properties for a 5xL matrix.
In cases where features are nonbinary (e.g. continuous or integers) it can be helpful for model performance to scale or normalize them. This will be discussed in the future.
For most models, their parameters are fit to minimize some objective function on the training data. The objective function varies depending on the type of model but they all involve the same idea of trying to get the predicted class to match the actual class. There are various optimizers which determine how the parameters are updated to minimize the objective.
There are many different classifiers available through scikit-learn, the python library for machine learning we will be using in this class. Some of the most commonly used classifiers are described below.
The class is decided by vote of the k nearest neighbors, where k is an integer. Sometimes these "votes" are weighted by distance so that closer neighbors have more influence.
Despite the name this is a classification method, not a regression method. It involves fitting a linear decision boundary based on a logistic class probability curve. It is inherently a binary classification method, the only way to turn it into a multilabel or multiclass method is by applying multiple logistic regressions.
Support vector machine classifiers tries to find a decision boundary that maximizes the margin of separation of classes across that boundary. The points on the boundary are the support vectors. You choose a kernel that defines the shape of the boundary, these can be nonlinear.
Decision trees split the data into segments with each split on the tree. The features are used to split the data (for example all data with a certain feature above a certain threshold will go on one side of the tree and the remainder on the other side). Splits are graded based on a user-chosen criterion for how well they separate the different classes. The tree continues for a defined depth or number of data points per leaf (the end of the tree branches). Classifications are based on the classes of the training points that get assigned to each leaf.
There are also multiple variations on decision trees. The main one is a random forest, which uses multiple trees. Each tree in the forest uses a random selection of features and classifications are determined by vote of the trees. Variations on random forests include extremely randomized trees (called extra trees in scikit learn) and XGBoost (which stands for eXtreme gradient boosting). These can often be more accurate than the baseline random forest. All tree-related methods can produce non-linear decision boundaries.
Hyperparameters are aspects of the model that can be defined by the user and that are not fit during training. These include things such as the depth of trees and number of trees in a random forest model. For models that have many hyperparameters, different combinations of hyperparameters may need to be tested to find the optimal combination. In scikit-learn this can be done using GridSearCV, which allows you to define a list of hyperparameters to test for each type of hyperparameter and then trains a model for each possible combination, and calculates metrics using cross validation. See the next section for the definition of cross validation.
Overfitting is when models are fit to noise or other non-general trends in the training data that will not be observed in practice. Because of overfitting models that have lower performance on the training data can sometimes outperform models with higher performance on the training set when applied in real world scenarios. Therefore, it is common to split data to get a better idea of model accuracy. Usually data is split three ways into training, validation, and test with most data being assigned to the training. The exact splits can vary but 80/10/10 is a common split. The validation set is used to choose the optimal hyperparameters. Often, cross-validation is used instead of having an initial validation set split from the training data. Cross validation involves taking a fraction of the data for validation, training on the remaining data, and then repeating the process with the next fraction being used as validation. For splitting the data into 5 fractions, this would be called 5-fold cross validation. Cross validation gives a better idea of how variable performance is with different training and validation data. Once a model and its associated hyperparameters have been chosen a final test of the model can be done using the test set. The test set should not be used before this because it might result in making a decision that artificially inflates test accuracy and makes it a worse measure of how the model will perform in practice.
It is also important to consider how the data was split. If there is highly similar data in the training and test datasets then performance may be artificially good on the test set. This is one form of data leakage. Common strategies to address this are to either further split the test set based on similarity to the training set to see how performance varies with training set similarity or to not use random splitting to avoid overlap between the training and test data. Usually this involves either splitting data in a way that the training and test are distinct (for example by first clustering data and making sure each cluster is entirely in one set) or by using a temporal split where older data is used as the training data and newer data is used as test data. This can still lead to overlap if the same or similar data has been collected twice but it does mimic how models may be used in practice and therefore may be the most realistic measure.
There are many different metrics that can be used to assess classification accuracy. Some of these are better than others at assessing specific aspects of the model's performance and so it is often helpful to consider multiple metrics. Many classification tasks have imbalanced data sets, where one label is much more common than another. For example, in high throughput screens for drug discovery there will always be many more inactive than active compounds. There are strategies to address this during model training but it is also important to consider it when looking at metrics. Some metrics, such as accuracy, can seem very high for imbalanced datasets. For example a model that predicts that all compounds are inactive will have high accuracy on high throughput screening data but is not very useful. Below are some common metrics and the associated equations:
Accuracy
Balanced accuracy (better than accuracy for imbalanced data sets)
TP is true positives, FP is false positives, FN is false negatives, etc.
Precision - how many of the things classified as positives are actually positive?
Recall (also called sensitivity) - how many of the positive things are found?
F1 score
Receiver operating characteristic (ROC) curve - plot of false positive rate (x-axis) vs true positive rate (y-axis). Can take the area under the curve to get a single number metric. 1 indicates perfect classification and a value around 0.5 is expected for random classification.
Precision recall curve (PRC) - plot of recall (x-axis) vs precision (y-axis), baseline performance for a random classifier will depend on the fraction of positives in the dataset. Like with the ROC, to get a numerical metric, you can also calculate the area under the PRC or area under the curve - area under the base line.
Example code for a chemistry example can be found in the github or accessed through the following Colab: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/3_Classifiers/3a%20Classifiers%20for%20Chemistry.ipynb
https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/3_Classifiers/3b%20Classifiers%20for%20Biochemistry.ipynb
You will need to upload the delaney csv file for the chem example and the full_promelt_seq.csv for the biochemistry example.