-
Notifications
You must be signed in to change notification settings - Fork 3
EmbeddingsMath
A set of XUnit tests showcasing how performing mathematical operations on embeddings can be used to reason about the relationships between words and phrases.
Adding 2 vectors together produces a new vector that is very close to the semantic meaning of the sum of the 2 original meanings. That is, "Monarch" + "Woman" is very close to "Queen" while "Monarca" + "Mujer" is very close to "Reina".
Subtracting 1 vector from another produces a new vector that is very close to the semantic, meaning of the "subtraction" of the 2 original meanings. As you might expect, operations such as addition and subtraction can be combined to produce new vectors that accurately represent performing those operations on the semantics of the text. Thus, "queen" - "woman" + "man" is very close to "king" while "king" - "man" + "woman" is very close to "queen".
Note: When performing multiple operations, it is best to only renormalize to unit-length at the end of all of the operations to avoid compounding error created by the normalization process.
The distance from a vector to the negation of that vector is, by definition, always ~2.0. Unfortunately, it is very difficult to reverse the process of embedding to understand what that new vector represents semantically. Based on the results of the Distance demos, it is clear that negation does much more than just create a semantic antonym since "king" and "queen" are actually fairly close in vector space, most likely because any two words that have that type of antoymn relationship are usually closely related in other ways. For example, both a "king" and a "queen" are monarchs, and are thus nearly synonyms in that context.
This test showcases the fact that we cannot identify which dimensions of the embedding encode the language of a particular word or phrase as nearly all dimensions are changed when comparing the embeddings of the same word (havinhg the same meaning) in two different languages. This is likely to be the same for other aspects of the text such as sarcasm, idiom, proverb, metaphor, etc.
This was an attempt to identify analogus relationships using vector subtraction. By subtracting "Puppy" from "Dog" and adding that vector to "Duckling", I had hoped to identify a vector close to "Duck". Unfortunately that doesn't seem to be the case, suggesting that the vector difference between puppy and dog does not capture the relationship between them in a repeatable way so we cannot supply that difference to other analogous relationships.
Ultimately, the vector relationship between "puppy" and "dog" cannot be used mathemtatically to capture other relationships in the embeddings space such as how "duckling" is related to "duck". The vector math to go from one to the other is not the same even though the relationships themselves seem to be the same, at least in one context.