This repository contains some work with SVM.
Some cool resources I have found about the topic
- https://research.wu.ac.at/en/publications/benchmarking-support-vector-machines-9
- https://www.cs.cmu.edu/~aarti/Class/10315_Fall20/lecs/svm.pdf
- https://www.cs.cornell.edu/courses/cs4780/2022sp/notes/LectureNotes13.html
- https://www.seas.upenn.edu/~cis5190/spring2019/lectures/07_SVMs.pdf
Graph of setosa (red) and virgina (blue)
| Linear | Polynomial |
|---|---|
 |
 |
| RDF | Sigmoid |
|---|---|
 |
 |
| 2D | 3D |
|---|---|
 |
 |
| Before | SVM Radial |
|---|---|
 |
 |
When the testing dataset size is (656, 2914) and training dataset is (164, 2914) with 4 different classes (4 different faces), a make_pipeline with RandomizedPCA(n_components=150) and SVC(kernel='rbf', class_weight="balanced"), we have a training time and accuracy of.
Training time: 4.62 Accuracy: 89.02%
| Unique Counts | Confusion Matrix |
|---|---|
 |
 |
Compared to a simple MLPClassifier with a hiddenlayer of 1024.
Training time: 6.31894588470459 Accuracy: 92.07
Now, what if we increase the dataset size?
The original analysis can be found in ./Downstream/textclassification.ipynb. Now, this dataset is much bigger compared to the facial recognition with a training set of (41157, 5000) for training and (3798, 500) and for testing.
This is in a sense a sentiment analysis problem, where tweets are given a sentiment of one of 5 classes whose distribution is shown below.
Inference time is also extremely slow.
The training time is extremely long.
Let's compare this to a simple linear regression classifier.
Here is the heatmap side by side.
| SVM Linear | Linear Regression |
|---|---|
 |
 |
In terms of accuracy, SVM does perform better. However, it is not worth the training cost.
Now let's compare these results to a simple LSTM recurrent neural network. It should be noted that training was also done on CPU.
In this example, we see that SVM can be used for regression tasks as well.
Here we select a couple of features.
This produces a mean squared error of
Here we add one other feature, Previous Close.
