Rescaling for KNN algorithm

[bensoltoff]

Replace weighted KNN with rescaling of variables? Or at least include a comparison of the methods.

[smiklin]

I was curious about it and played around with the scaling. I am not sure if I did the right/best thing—I tried two options, one in which I scaled the train and the test set separately, the other when I scaled both using the full data mean and sd for each variable (except the outcome). The resulting graph looked pretty much the same. Plot bellow is using the alter scaling

Also, I think I know what went wrong with the weighed KNN plot—it has that huge error. It's because part of the code kept some variables from a previous example

mse_knn <- tibble(k = c(1:10, seq(20, 100, by = 10)),
                  knn = map(k, ~ kknn(biden ~ .,
                                      train = biden_train, test = biden_test, k = .)),
                  mse = map_dbl(knn, ~ mean((sim_test$y - .$fitted.values)^2)))

That last line should be
mse = map_dbl(knn, ~ mean((biden_test$biden - .$fitted.values)^2)))

[bensoltoff]

Thanks for the head's up on the original plot. A correction is in the works now for the original plot.

I'm still not satisfied from a theoretical standpoint why the two methods (weighted vs rescaled) seem to generate the same conclusions. One note on rescaling the variables with a training/test split. Like we talked in class today, you need to avoid data leakage. That is, your test set cannot be used in any way to construct and fit the model. Otherwise your estimate of the test MSE is biased. The typical approach is to rescale the variables during feature engineering. So the general process would be something like:

1. Randomly partition dataset into training/test sets
2. Construct new/modified features. In this case, that would mean to standardize the variables in the training set based only on training set observations.
3. Subdivide the training set into training/validation sets using validation set/LOOCV/k-fold cross validation
4. Fit the models using the (smaller now) training set
5. Evaluate models using the validation set
6. Once you choose a final model, use the test set to evaluate its performance. However, you need to first apply the same feature transformations to the test set that you did to the training set. So you need to save all the values used to mean-shift and standardize the original variables. For instance, say you originally standardized x1 by mean-centering at 0 and dividing by its standard deviation. You need to save the original mean and standard deviation from the training set, then use those values to transform the test set. So if the training set x1 mean was 4 and standard deviation was 2, in the test set subtract 4 from x1 and divide by 2. Don't use the mean and standard deviation calculated from the test set because that was not the original transformation.

[smiklin]

@bensoltoff it really does make the most sense to rescale the test set using the train mean and sd, somehow I didn't even examine it as an option.
I did start working on using the 10-fold cv but I'm still working my way through it. For the sake of simply comparing scaled and weighed KNN (if we're sticking with the biden set), would I still want to hold out a test set, or would it make more sense to just use the whole set (the way you did with the cars for the class example)? I can easily run both, but I was just wondering

[bensoltoff]

Probably easier to ignore the test set. Since we want estimates of the error for different values of k, we would be reusing the test set anyways which defeats the whole point of it.

[bensoltoff]

Plus that data is simulated, so we could calculate the true test error rate from the data generating process

[smiklin]

The biden data is simulated? I think I am a bit disappointed now...

I gave it another go, using the whole data set and 10-fold cross-validation. Rescaled each fold using the mean and the median of the whole data set.

The plot still looks the same—here's a bit more of a close-up view:

But still, this doesn't get to the theory? Out of curiosity, I also did a scaled weighed plot, and it's pretty much the same as the weighed plot? Let me know if this is something you'd have time/interest to discuss in office hours.

[bensoltoff]

Oh no. The Biden data is real. The wage data is simulated.

…

On Tue, Jan 29, 2019, 5:00 PM Sanja Miklin ***@***.*** wrote: The biden data is simulated? I think I am a bit disappointed now... I gave it another go, using the whole data set and 10-fold cross-validation. Rescaled each fold using the mean and the median of the whole data set. The plot still looks the same—here's a bit more of a close-up view: [image: screen shot 2019-01-29 at 16 51 06] <https://user-images.githubusercontent.com/30011000/51946198-742bf980-23e6-11e9-8245-0232b6599dcf.png> But still, this doesn't get to the theory? Out of curiosity, I also did a scaled weighed plot, and it's pretty much the same as the weighed plot? Let me know if this is something you'd have time/interest to discuss in office hours. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF7gq2Auls9la0rPiOxIZ7PR-goSxYOmks5vINKXgaJpZM4aPqyL> .

[bensoltoff]

The results look interesting. We can talk more in office hours. I'll also do some more digging into the differences between the two methods.

…

On Tue, Jan 29, 2019, 5:00 PM Sanja Miklin ***@***.*** wrote: The biden data is simulated? I think I am a bit disappointed now... I gave it another go, using the whole data set and 10-fold cross-validation. Rescaled each fold using the mean and the median of the whole data set. The plot still looks the same—here's a bit more of a close-up view: [image: screen shot 2019-01-29 at 16 51 06] <https://user-images.githubusercontent.com/30011000/51946198-742bf980-23e6-11e9-8245-0232b6599dcf.png> But still, this doesn't get to the theory? Out of curiosity, I also did a scaled weighed plot, and it's pretty much the same as the weighed plot? Let me know if this is something you'd have time/interest to discuss in office hours. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF7gq2Auls9la0rPiOxIZ7PR-goSxYOmks5vINKXgaJpZM4aPqyL> .

[smiklin]

I took a fresh look at this, and I think I figured a part of it out. It turns out that the kknn() function has an argument scale which defaults to T. A more informative plot (100 simulations for each k for each model, using 10-fold cv):

The question then becomes: why does a scaled-only model seem to do slightly better than a scaled-weighted model? Is it a characteristic of the modeling strategy or of this particular data set?

let me try a completely different data-set and see what I get. TBC

Edit: I 'scaled up' age for the example above, multiplying it by 10,000 to see better what is happening when a variable is far out of scale

[bensoltoff]

Yes, I see now. Most of the k-nearest neighbor functions in R (and likely Python) auto-rescale the variables before estimating the model. Same thing with other methods that require scaled variables (e.g. principal components analysis). I need to add a more explicit discussion of this for future years.

[smiklin]

OK, I simulated (did my best to? I don't trust myself 100% on this one, but there is a dataset) another Biden dataset, that includes age, income, democrat and female. Income is, therefore, of a much different scale than the previous two. I did only 10 simulations for each this time because of time.

and a close-up:

This one makes a it more sense to me, because as k increases, the weighted does better which is to be expected? (I did random sampling for all variables for my data)

I'm wondering ultiamtely if the type of scaling matters and interacts with the model in possibly problematic ways? E.g. if the assumption of the enar-normal distribution is violated?