SGDClassifier breaks down when encountering unseen values?

Describe the bug

https://stackoverflow.com/questions/70253946/sgd-breaks-down-when-encountering-unseen-values

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

[ogrisel]

Is this specific to SGDClassifier? Or can you reproduce the problem with other classifiers with different solvers such as LogisticRegression?

Can you please craft a minimal reproducer with a synthetic dataset so that we can reproduce the problem without any ambiguity? Please also include the expected and observed results in your report.

[Nivi09]

Hello, I would like to work on this. As ogrisel mentioned, can you provide the synthetic dateset to reproduce the problem ?

Hello @ogrisel , @Nivi09 - thank you for your answers.

I do not have any synthetic data, I use a real dataset that I cannot really share.

But if you pick any dataset from the Internet that has categorical variables or even better text then you can easily test what I describe.

It does not have to be only eg with categorical; it can be 9 numerical features and 1 categorical feature.
Especially if the categorical feature has some importance (although the same may happen if it does not; I have not tested that) and in your test set one observation has one unseen value for the categorical feature then the SGD results returns essentially random predictions (or at least not directly related).

It may be that scikitLearn implementation is fine but just SGD itself has this weakness in general?
I think though that RandomForestClassifier does not have the same problem in these cases.

I have not tested much the LogisticRegression.
even like that:
logistic_regression = LogisticRegression(penalty='elasticnet', tol=10, C=1.0, class_weight='balanced', random_state=0, solver='saga', max_iter=20, n_jobs=os.cpu_count()-1, l1_ratio=0.7)
it takes too much time to converge

[Nivi09]

Hi @GeorgeZan , let me rephrase the issue that you have described and please correct me if I have not understood it correctly.
The task is a classifier algorithm, let's say a data set with 9 numerical predictors and target label as class. For example, let's say you have dataset of len 1000 and your target label contains 5 classes - C1, C2, C3, C4, C5 and there is only one observation for class C5. When you divide it for train and test, your train and test split algorithm has divided the classes C1, C2, C3, C4 divided in some proportion in both train and test, but because observation of class C5 is only one it gets allotted to test set. Now, when you train SGD classifier on train set and then test it on test set, the classifier breaks down for class C5 . Is my understanding correct ?

[ogrisel]

The task is a classifier algorithm, let's say a data set with 9 numerical predictors and target label as class. For example, let's say you have dataset of len 1000 and your target label contains 5 classes - C1, C2, C3, C4, C5 and there is only one observation for class C5. When you divide it for train and test, your train and test split algorithm has divided the classes C1, C2, C3, C4 divided in some proportion in both train and test, but because observation of class C5 is only one it gets allotted to test set. Now, when you train SGD classifier on train set and then test it on test set, the classifier breaks down for class C5 . Is my understanding correct ?

In general, any classifier can never predict a class that was not part of the training set. I am not sure what "break down" means in this context.

But if you pick any dataset from the Internet that has categorical variables or even better text then you can easily test what I describe.

Can you please craft a standalone reproducible code snipet example that generates its own toy dataset so that we can just execute ? I am afraid the English language is ambiguous while Python is not.

Please also report the result you observe and the output you would have expected.

[Nivi09]

Hi @ogrisel, GeorgeZan has mentioned that - SGD results returns essentially random predictions (or at least not directly related).
I think this is what break down means. Please correct me if my understanding is not correct @GeorgeZan ?

And as ogrisel mentioned, my understanding is also the same that no classifier can predict a class that was not a part of training set.

I read GeorgeZan comment again, so here is my second understanding :
input set X has 10 independent variables in which one variable is categorical. Along with that you have categorical dependent variable. Your test fails when there is a class in categorical I.V. which has not been seen in train data set while training and that's where you get random prediction.

@GeorgeZan, please let me know, out of two examples, which one is correct.

@Nivi09, yes indeed.
"Hi @ogrisel, GeorgeZan has mentioned that - SGD results returns essentially random predictions (or at least not directly related).
I think this is what break down means."

To give a hypothetical example, let's say I have a model with 9 numerical features and a categorical feature and let's say there are 3 classes in my problem (A, B, C class).
Let's also assume that I have a test set instance X that belongs to the A class.
Specifically, it belongs to class A regardless of the value of the categorical feature.
At best, the categorical feature can help the model to predict class A for X just by inference but it is not a defining feature of X.
i)
If I keep the categorical feature in the model and the test set instance X provided has a known/seen value at the categorical feature then the model returns the following prediction:
A: 0.7
C: 0.2
B: 0.1

iI)
If I drop the categorical feature from the model then the model returns the following prediction:
A: 0.55
C: 0.35
B: 0.1

iii)
If I keep the categorical feature in the model and the test set instance provided has a unknown/unseen value at the categorical feature then the model returns the following prediction:
B: 0.5
A: 0.3
C: 0.2

.........................
Obviously the impact of the categorical features in all these cases depends on how important is the feature for the model; in the example above it is assumed that the categorical feature has quite significant weight.

This is also why in (ii) the top score drops by quite a lot in comparison with (i) given that it is only one feature.

However, and this is what I mean that the model "breaks down", in (iii) it returns essentially random predictions.
It does not make sense to me that if you just receive an unseen value your prediction changes so so much.
I would expect (iii) to be more like (ii).

The even more interesting and problematic thing is that the same thing happens as mentioned at my original post if you have eg 100k tf-idf sparse features and just one categorical at the same model and if you put an unseen value at the categorical at the test set then SGD "breaks down" as in (iii).

In the dataset that I experimented the categorical feature is quite significant (eg dropping it reduced the overall accuracy of the model by 5 percentage points; eg accuracy drops from 80% to 75%) but I would not exclude based on how unstable SGD is that the same thing would happen even if the feature is not that significant.
.........................
@ogrisel I think now it must be quite clear.

As I said, you can pick any dataset on the Internet which has also categoricals or you can even create a very basic one on Excel and I think you will observe what I mean.

[Nivi09]

Hello @GeorgeZan , for the case 3 as you have mentioned:

If I keep the categorical feature in the model and the test set instance provided has a unknown/unseen value at the categorical feature then the model returns the following prediction:
B: 0.5
A: 0.3
C: 0.2

So, to which class do you expect this example to belong to ? Because I'm confused here when you mentioned - - ' (iii) it returns essentially random predictions.
It does not make sense to me that if you just receive an unseen value your prediction changes so so much.
I would expect (iii) to be more like (ii).'

@Nivi09 ah sorry, just edited my post to add "Let's also assume that I have a test set instance that belongs to the A class.".
Although it is relative obvious what I mean.

My point is that if you have a known value at the categorical feature or a if you entirely drop the categorical feature then the model returns the right prediction (although with slightly different "confidence").

But when you put an unseen value at the categorical feature then the model returns random results although given (ii) essentially the categorical feature does not play much role in the prediction so (iii) should have been returning something more like (ii).

[glemaitre]

I don't think that there is anything wrong but I would not be able to be sure until we have an example showing the wrong behaviour and the expected behaviour.

Since SGDClassifier is a linear model, when it comes to categorical data, the type of encoding is rather important. So did you check the coefficients of your model? Did you use an OneHotEncoder for encoding your categories? Basically, the value of the "unseen categories" will have an impact depending on the coefficient of the associated variable.

@glemaitre as I mention at my post, I put the categorical value in the text and then apply tf-idf on it.
In this sense you could say that I do one hot encoding on it (?).

I have not really used the categorical variable separately but when I did was with label encoder.
Even though not ideal because "LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat.", still it should not break down so much with unseen values.

PS
"I don't think that there is anything wrong "
Wrong or not, the question is why this happens like that with SGD and unseen values.