Repository navigation
Ch 5 Feature Engineering and Dimensionality Reduction
Models generally perform better if all features fall into similar ranges. Data can be preprocessed to accomplish this and scikit-learn has a few different scaler that can accomplish this.
- Standard: scales data to a mean of 0 and variance of 1
- Min max: scales all data to 0 to 1 range
- Max absolute value: scales all data to a -1 to 1 range
- Power scaler: fit data to a normal distribution
- Normalize: scale so data has a norm of 1
- Robust scaler: scales data according to quantile range, robust to outliers
Which scaler is best depends on your data, for example if it has outliers.
When we have a lot of features, or in other words, when our features occupy a high dimensional space, overfitting becomes more likely. This is because, for most ML models, each feature will be assigned a weight/coefficient so more features mean more weights which makes a model more capable of overfitting. Therefore, to avoid overfitting it may help to reduce the number of features or in other words reduce the dimensionality of our features. One way to do this is by selecting certain features that are not predictive or that are redundant with other features to discard, this is part of feature engineering that will be discussed below.
Another common way to reduce dimensionality is by principal component analysis (PCA) which is a mathematical method to reduce the number of dimensions in data. PCA involves less expert knowledge or choices made by the model developer than some other types of feature engineering. PCA transforms or rotates data into a new coordinate system, with the principal components being the new coordinates. These components are ordered by how much they contribute to the overall variance of the data. Therefore, in ideal scenarios, by choosing the top few components we can capture most of the information contained across all features in much fewer dimensions. How much information is lost by doing this will depend on the specific dataset. The explained variance of the chosen components can be used to determine how much variance is captured by the chosen number of dimensions. Note that PCA is an unsupervised technique, which will be discussed more in future lectures. In some cases the features that contribute most to the variance may not be the same ones that are predictive of the label in supervised learning, and PCA will not work as well because a lot of the information contained in that feature may be lost. Analysis of PCA loadings (how much each feature contributes to each PC) can help determine which features are actually being used in a model trained on PCA transformed features. PCA is sensitive to the range of values of features so it is important to preprocess features so that they are scaled before using PCA. Preprocessing and PCA can be linked together with pipelines in scikit-learn.
PCA is also useful for visualizing data. We will generally have more than three features, and so it will not be possible to plot the data across all features. With PCA we can reduce the data to two or three dimensions to visualize the data. PCA is a linear transformation of the data, with many data points and many features it becomes less likely that PCA will cleanly separate data that falls into different classes. Because of this some nonlinear transformation methods like tSNE and UMAP have also become popular, when these methods are used to transform data into two dimensions separation between groups in the data become more clear, but disadvantages of these methods are that the global structure that emerges is not trustworthy and sometimes non-meaningful clusters emerge. Therefore tSNE and UMAP are only used for visualization and not for reducing the number of features. Interpretations made from these visualizations should be treated with caution. We will cover how to use tSNE and UMAP during the unsupervised learning lecture.
Example code for feature scaling and dimensionality reduction:
Chemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/5_Feature_Engineering_and_Dimension_reduction/Chem_Dim_Red.ipynb
Biochemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/5_Feature_Engineering_and_Dimension_reduction/Biochem_Dim_Red.ipynb
Example code for feature engineering and supervised dimensionality reduction:
Chemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/5_Feature_Engineering_and_Dimension_reduction/Chem_Feature_Engineering.ipynb
Biochemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/5_Feature_Engineering_and_Dimension_reduction/Biochem_Feature_Engineering.ipynb