en_core_web_trf: no word vectors loaded

[BramVanroy]

Hi there, me again. I've been looking through the documentation but still don't understand. In the base Transformer models, a transformer model is present as the shared underlying model. This implies that it replaces tok2vec from other models. However, I would still expect that this transformer component adds word vectors - which is not the case.

nlp = spacy.load("en_core_web_sm")
nlp_trf = spacy.load("en_core_web_trf")
nlp_lg = spacy.load("en_core_web_lg")
nlp.pipe_names
Out[7]: ['tok2vec', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer']
nlp_trf.pipe_names
Out[8]: ['transformer', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer']
nlp_lg.pipe_names
Out[9]: ['tok2vec', 'tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer']

When I calculate the similarity between tokens with token1.similarity(token2) when using an underlying Transformer model, I get the warning:

UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. en_core_web_sm, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.

Additionally I also get the warning that I am evaluating on empty vectors.

Wouldn't transformer models be perfect to get contextualised word vectors? Why are they not included, or how does that differ from the "regular" models?

[adrianeboyd]

All the necessary information is in doc._.trf_data, but since different applications may want to calculate the vectors in different ways, we don't set anything by default. The alignment for the strided spans used in en_core_web_trf also requires a bit of additional understanding of the internal details. Some pointers to relevant discussions:

• Transformer based word vectors #6511 (comment)
• TRF Alignment wrong? #7032