Dimensionality Reduction

Defination

Dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature extraction.

• feature selection: select a subset of the original feature set.
• filter: meature the importance of each feature，ranking，and select top x%
  • PCA/FA/LDA
• wrapper: evaluate each subset, keep good delete bad, re-conbine, so on so forth.
  • SVM
  • RandomForest
• embedded:
  • forward and backwards selection
  • Lasso
  • Elastic Net
• feature extraction: build a new set of features from the original feature set.

What is the difference between feature selection and dimensionality reduction?

Feature selection yields a subset of features from the original set of features, which are best representatives of the data. It is an exhaustive search. Dimensionality reduction transforms the original feature space to yield a subspace, but the basis of the subspace cant be interpreted as a particular feature. In text data, features might be size of characters or some global features of the text. Feature selection will keep only certain features of those. Dimensionality reduction will give you a geometric transform that will give you 'features' which cannot be easily interpreted as say, size or vertical projections, etc. This technique might reduce the correlation or higher order statistics between the features by representing in a different way.

Methods

Linear:

• PCA (Principal Component Analysis)
  • PCA principle
• MDS (MultiDimensional Scaling)
• LDA (Linear Discriminant Analysis)
  • Supervised dimensionality reduction

Non-Linear:

• ISOMAP (Isometric Feature Mapping)
• LLE (Locally Linear Embedding)
• LE (Laplacian Eigenmap)
• LPP (Locality Preserving Projections)

Reference:

• Multiple dimensional reduce method
• Very good resource:combine Scikit-learn introduced some method