Use model's own embedding to compute similarity smart tags.

[Dref360]

Hello!

It would be useful to me if I could use the model's own embedding to compute similarity smart tags. For my particular use case, semantic embeddings are not useful.

Proposal

config.json

{
	"similarity" : {"faiss_encoder": "model" } // could be 'self-similar'? 
}

One can get the embedding of their HuggingFace model with:

inputs = tokenizer(...)
model : BertModelForSequenceClassification = ...
embedding = model.base_model(**inputs).last_hidden_state[:, 0, :] # Take first token embedding

A simpler approach is to load the same model with the feature-extractor task, but that might be more involved.

pipe = pipeline('feature-extraction', model='your_model', truncation=True)

EDIT: One can also do

pipe2 = pipeline('feature-extraction', model=pipe.model.base_model, tokenizer=pipe.tokenizer)
``` where pipe is the initial PipelineForSequenceClassification.

[gabegma]

Hey @Dref360 - that's a great suggestion, we've had this request a few times. We'll aim to prioritize that in the coming weeks.

[Dref360]

I didn't realize that sentence-transformer can now load model directly from the hub and convert them. The only caveat is that it uses mean pooling instead of the first token. Not a terrible issue AFAIK.

So if your model name is cardiffnlp/twitter-roberta-base-sentiment-latest, Sentence Transformers will extract the Roberta base when you do

model = SentenceTransformers('cardiffnlp/twitter-roberta-base-sentiment-latest').

It also uses authentication so it works with private repo as well.

Closing!