Mining the geometry of embedding spaces

[thorwhalen]

Possible R&D projects for teasing out the relationship between embeddings space geometry and linguistic properties.

[thorwhalen]

relational vector mining

The queen ~= king - male + female is really pointing to the fact that king - queen, male - female, boy - girl should are be approximately equal. We call this vector they are all close to, the "relational vector".
Note the difference between semantic direction and vector.

Question: What other relation vectors can be mined out of a fixed given embedding space?

I can think of two immediate unsupervised versions of inquiring into this question: One where it's a word-to-word relationship (like king and queen), and another is a words-to-words relation ship (like the set {"king", "queen", "prince", "duke", "monarch", ...} and {"peasant", "serf", "commoner", "villager", "laborer", ...}.

one-to-one relationships

The brute force solution is scalable up to a certain point. Just take the matrix of all embeddings of a given list of n words, then compute the difference of every word with the other words. You now have n (n-1) word differences, and you can do clustering algorithms (with bounded distance) on that to find difference vectors that are all close to each other.

The the brute force works, we probably still want to find ways to shave off a lot of the computation (and memory!) to make it more scalable.

many-to-many relationships

In linguistics, when two sets of words are semantically opposed as groups but not individually (e.g., "royalty words" versus "commoner words"), this relationship is often referred to as contrastive lexical fields or oppositional semantic fields. This phenomenon aligns with Lexical Field Theory, which suggests that fields form closed sets with no overlaps or gaps between lexemes. Additionally, it can be described as a form of binary opposition or contrastive opposition, depending on the context.

Of course, a brute force would work here too, but here the limits of scalability are quickly reached, since it's exponential (iterating over sets of sets).

[thorwhalen]

Dealing with Homonymy in High-Dimensional Embedding Spaces

The Problem: Homonymy and Polysemy in NLP

In natural language, the same word can have multiple meanings depending on context. This phenomenon falls under two main linguistic categories:

• Homonymy (same form, different meanings, unrelated)
  • Example: "bat" (the flying mammal) vs. "bat" (used in baseball).
• Polysemy (same form, different meanings, but related)
  • Example: "bank" (financial institution) vs. "bank" (side of a river).

These ambiguities exist within a single language but also across languages (e.g., "chat" means "to talk" in English but "cat" in French). This cross-lingual case can be thought of as false friends—words that look similar across languages but have different meanings.

How Embeddings Solve This Problem

Traditional NLP models, such as Word2Vec or GloVe, assigned a single vector per word, meaning "bat" (animal) and "bat" (sports) shared the same embedding, causing confusion in downstream tasks.

However, modern contextual embeddings (from transformers like BERT, GPT, T5) solve this problem by dynamically adjusting word representations based on context.

High-Dimensionality and Multiple Meanings

Embeddings exist in thousands of dimensions (e.g., 1,536 or 3,072 for OpenAI models). This allows:

• Contextual Separation: "bat" can be close to "mammal" in animal-related dimensions but close to "baseball" in sports-related dimensions.
• Cross-Lingual Alignment: "chat" (French: "cat") is near "cat" in pet-related dimensions, while "chat" (English: "talk") is near "conversation" in communication-related dimensions.
• Multiple Similarities at Once: A word can be similar to two different words in different directions without distortion.

In low-dimensional spaces, forcing "chat" near both "cat" and "talk" would distort the space. In high-dimensional embedding spaces, this isn't a problem—there are enough dimensions to encode all relationships independently.

Research Tasks: Investigating Homonymy in Embeddings

1. Quantifying Context Sensitivity:

   • Measure how a word’s embedding shifts in different sentences.
   • Example: Compare "bat" embeddings in "A bat flew overhead" vs. "He swung the bat."

2. Cross-Lingual Homonymy Detection:

   • Find words that have different meanings across languages but share the same form.
   • Analyze their embeddings to see if disambiguation naturally emerges.

3. Dimensional Analysis of Homonyms:

   • Identify which embedding dimensions encode different senses of a word.
   • Apply PCA (Principal Component Analysis) to visualize separations.

4. Probing Specific Transformer Layers:

   • Extract embeddings from early vs. late transformer layers to see where meaning separation happens.
   • Hypothesis: Lower layers retain word identity, middle layers separate meanings, and final layers specialize based on task.

5. Evaluating Multi-Sense Embeddings in Downstream Tasks:

   • Test if contextualized embeddings improve machine translation and semantic search.
   • Example: Check if "chat" in "je caresse mon chat" translates correctly to "cat" but "nous avons un chat" translates to "talk".

By exploring these areas, we can better understand how deep learning models handle homonymy and improve their ability to disambiguate meaning in both monolingual and multilingual settings. 🚀

[thorwhalen]

The (counterintuitive) properties of high-dimensional spaces

In high-dimensional spaces, many intuitive properties of geometry—such as distances, angles, and volume distribution—behave in unexpected ways. These effects influence how machine learning models, especially NLP embeddings, represent and compare words. The table below summarizes key high-dimensional phenomena, their mathematical effects, and their impact on NLP tasks like similarity search and clustering.

Phenomenon	Effect in High Dimensions	Impact on NLP
Curse of Dimensionality	All distances become similar	Nearest-neighbor search becomes harder
Volume Concentration Near the Surface	Most volume is near the surface of high-dimensional shapes	Cosine similarity works better than Euclidean distance
Cosine Similarity is More Stable	Angles are more meaningful than distances	Used for semantic similarity in embeddings
Concentration of Measure	Almost all vectors have similar lengths	Magnitude is less important than direction
Hollow Ball Effect	Most mass is near the surface	Word vectors lie on a high-dimensional shell

Links

• Surprises in high dimensions