add option to normalize embeddings

[PhilipMay]

Add option to normalize embeddings.

[PhilipMay]

@blakechi do you have an idea how to best add this to the differentiable head ?

Somewhere here:

setfit/src/setfit/modeling.py

Line 232 in 5313939

outputs = self.model_body(features)

Can we just change this:

                    outputs = self.model_body(features)

to

                    outputs = self.model_body.encode(features, normalize_embeddings=self.normalize_embeddings)

?

[blakechi]

Hi @PhilipMay,

Sorry for the late reply.
I notice that sentence-transformers's encode function sets the model to eval mode at here.
Since users might want to train SetFitModel in the end-to-end fashion, I think it's not a good idea to use encode. Instead, we might need to normalize embeddings by ourself.

It's not totally the same case but I tried adding a LayerNorm between the body and head before. And I found that it degraded the performance. Just for reference. 😀

[PhilipMay]

@blakechi what do you think about the norm. solution I added to this PR? Can you please check it?

See this commit: 88d7712

I just stole it form sbert. See:

https://github.com/UKPLab/sentence-transformers/blob/0b5ef4be93d2b21de3a918a084b48aab6ba48595/sentence_transformers/SentenceTransformer.py#L183-L184

[blakechi]

It looks good to me :)

[blakechi]

Just curious, have you tested the performance? Or do you have other purposes?

[PhilipMay]

Just curious, have you tested the performance? Or do you have other purposes?

I tested performance with the "normal" Logistic Regression from SKlearn.
Normalizing reduces performance - which is surprising for me. Also see here:

#135 (comment)

@blakechi do you have an idea why?

PS: To test the diff. head is still on my todo list.

[PhilipMay]

Can be merged from my point of view.

[blakechi]

Just curious, have you tested the performance? Or do you have other purposes?

I tested performance with the "normal" Logistic Regression from SKlearn. Normalizing reduces performance - which is surprising for me. Also see here:

#135 (comment)

@blakechi do you have an idea why?

PS: To test the diff. head is still on my todo list.

I don't have a clear answer. But my guess is the magnitude of the embeddings might contains some information that helps classification, like the occurring frequency of each words?
An interesting report.

Btw, is it better to open embedding normalization to users only when we find merits from it?

[PhilipMay]

Btw, is it better to open embedding normalization to users only when we find merits from it?

I would say that ML is a field where experimentation needs to be done. You never know what exactly helps and what not.
It also depends a lot on the use-case & data. So this enables ppl. to experiment. Since we disable it by default we do no harm. We just add one more option to experiment.

[lewtun]

Thanks for adding this option @PhilipMay ! Since it doesn't interfere with the existing training, I'm happy to include it :)

[blakechi]

I would say that ML is a field where experimentation needs to be done. You never know what exactly helps and what not. It also depends a lot on the use-case & data. So this enables ppl. to experiment. Since we disable it by default we do no harm. We just add one more option to experiment.

Good point, that's what I didn't notice and it makes sense to me!
Thanks for letting me know 😄 @PhilipMay