The geometry of meaning: Semantic Embedding Transformations Research

[thorwhalen]

Quick links:

• Geometry of Meaning chatbot.
• Podcast Episode - The Geometry of Meaning: Word Analogies

Semantic Embedding Transformations Research

Research on discovering linear relationships within embedding spaces focuses on systematically identifying vectors or directions that correspond to meaningful linguistic transformations. This approach has been studied in the context of word embeddings such as Word2Vec and GloVe, as well as in more advanced, contextual models like BERT or GPT. In these embedding spaces, certain linear transformations often encode grammatical or semantic shifts, such as moving from singular to plural forms, changing tense, or flipping a conceptual property like gender. One of the most widely cited examples is the analogy king – male + female ≈ queen, which suggests that an axis in the embedding space represents a gender transformation.

Within the framework of natural language processing (NLP) and machine learning (ML), it is useful to think of these embeddings as points in a high-dimensional distributional space, where semantic or syntactic similarities emerge as geometric regularities. The discovery of specific “semantic directions” in the space can be treated as a problem in subspace identification, matrix factorization, and interpretability. From a deep learning perspective, the challenge lies in separating meaningful signals from random noise in high-dimensional embeddings, especially when the models are trained on large corpora and contextual transformations (e.g., via Transformers) introduce additional complexity.

From a linguistic point of view, these directions often correlate with classical morphological and syntactic categories, such as number, tense, aspect, and gender, reflecting the morphological and grammatical structures captured by the model. Research into typological variation is also relevant, as different languages might encode such distinctions differently, or not at all, raising interesting questions about cross-lingual embedding alignment and how to isolate or disentangle shared linguistic features.

A key line of research investigates whether these embedding directions are robust and consistent across multiple language models and corpora. This links to broader questions of interpretability and bias: for instance, identifying and mitigating unwanted stereotypes or biases embedded in these directions. Other methodological areas include probing tasks to test whether a model’s latent representations capture certain linguistic properties, as well as designing specialized losses or regularization techniques that encourage embeddings to reflect particular conceptual dimensions.

Numerous research topics arise at the intersection of these ideas. For example, subspace alignment techniques in multilingual NLP explore whether similar transformations emerge across languages and whether they can be mapped or transferred. In interpretability studies, linear transformation analyses aim to locate inherent biases in the geometry of embeddings. Meanwhile, advanced factorization or decomposition methods (PCA, SVD, or more sophisticated manifold learning approaches) are used to extract principal axes corresponding to salient linguistic and semantic distinctions.

Relevant keywords for searching the literature include:
• word embeddings
• semantic directions
• linear transformations in embeddings
• linguistic interpretability
• distributional semantics
• word analogy tasks
• embedding geometry
• debiasing word embeddings
• morphological transformations
• cross-lingual embedding alignment
• subspace identification
• probing classifiers
• contextual embeddings (BERT, GPT)
• bias and fairness in NLP
• manifold learning in representation spaces
• interpretability and explainability in deep learning

Such terms often appear in the technical literature and conference proceedings related to NLP (e.g., ACL, EMNLP, NAACL) and ML (e.g., NeurIPS, ICML, ICLR).

[thorwhalen]

Literature for our research

Semantic Directions

• Decoding Word Embeddings with Brain-Based Semantic Features: This study examines the interpretability of word embeddings by aligning them with semantic features derived from brain activity, aiming to understand the semantic information encoded within embedding dimensions.

• From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models: A comprehensive review of the evolution from word embeddings to multimodal embeddings, discussing techniques, applications, and future directions in the context of large language models.

• SeNSe: Embedding Alignment via Semantic Anchors Selection: This paper introduces SeNSe, a method for aligning word embeddings across different vector spaces by selecting semantic anchors, facilitating tasks like machine translation and transfer learning.

• Enhancing Word Embeddings for Improved Semantic Alignment: The study presents a method to enhance word vectors by incorporating richer semantic properties, improving alignment with semantic reference resources such as WordNet.

• Studying Word Meaning Evolution Through Incremental Semantic Shift Detection: This research explores methods for detecting semantic shifts over time using word embeddings, providing insights into how word meanings evolve incrementally.

• A Detailed Review on Word Embedding Techniques with Emphasis on Word2Vec: A comprehensive review of various word embedding techniques, with a focus on Word2Vec, discussing their applications and effectiveness in capturing semantic relationships.

• Word Embeddings: A Survey: This survey provides an overview of the main strategies for building word embeddings, highlighting their ability to encode syntactic and semantic information useful for various NLP tasks.

• Combining Semantic Matching, Word Embeddings, Transformers, and LLMs for Efficient Information Retrieval: The paper discusses the integration of semantic matching, word embeddings, transformers, and large language models to enhance information retrieval processes, addressing challenges in handling large volumes of scientific publications.

• Lexical Semantics Enhanced Neural Word Embeddings: This work introduces methods to enhance neural word embeddings by incorporating lexical-semantic relations, aiming to improve the representation of semantic similarity and relatedness.

• Diachronic Word Embeddings and Semantic Shifts: A Survey: A survey of research related to diachronic word embeddings and methods for detecting semantic shifts over time, discussing various approaches and their effectiveness.

linear transformations in embeddings

• Word Embeddings Are Steers for Language Models: This paper explores how linear transformations of word embeddings can steer language model generation styles, introducing a method called LM-Steer that applies a learnable linear transformation to output word embeddings for flexible and transparent control over language model outputs.

• LM-Switch: Lightweight Language Model Conditioning in Word Embedding Space: The study presents LM-Switch, a technique that inserts a learnable linear bias into the language model word embeddings, enabling efficient conditioning of language models by adjusting word embedding transformations.

• Systematic Evaluation of Linear Transformations for Cross-lingual Semantic Spaces: This research systematically evaluates linear transformation methods for aligning cross-lingual semantic spaces, providing insights into the effectiveness of various approaches in mapping word embeddings across languages.

• Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space: The paper investigates how word relationships can be represented as orthogonal transformations within embedding spaces, demonstrating that analogical reasoning can be performed using such transformations.

• Analogies Explained: Towards Understanding Word Embeddings: This work provides a probabilistic explanation for the linear relationships observed in word embeddings, offering insights into why certain word analogies form parallelograms in embedding spaces.

• Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation: The study reveals that word embeddings capture divergent linguistic information and that linear transformations can adjust similarity orders, enhancing performance in specific linguistic aspects.

• Learning Linear Transformations between Counting-based and Prediction-based Word Embeddings: This research proposes a method for learning linear transformations between counting-based and prediction-based word embeddings, facilitating the prediction of distributed word embeddings for novel unseen words.

• Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations: The paper introduces a multi-step framework of linear transformations that generalizes previous work on bilingual word embedding mappings, improving performance in zero-shot bilingual lexicon extraction.

• On the Reproducibility and Generalisation of the Linear Transformation Approach to Cross-Language Embedding Alignment: This study examines the reproducibility and generalization of linear transformation approaches for cross-language embedding alignment, validating their effectiveness in multilingual scenarios.

• Analyzing the Limitations of Cross-lingual Word Embedding Mappings: The research analyzes the limitations of linear transformation methods for cross-lingual word embedding mappings, questioning the isomorphism assumption and suggesting directions for improvement.

Linguistic Interpretability

• eXplainable AI for Word Embeddings: A Survey: This survey provides a comprehensive review of eXplainable AI strategies focused on enhancing the interpretability of word embeddings, highlighting the need for transparency in AI systems. Springer Link

• SensePOLAR: Word Sense Aware Interpretability for Pre-trained Contextual Word Embeddings: This paper introduces SensePOLAR, an extension of the POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings, addressing the challenge of polysemy in embeddings. arXiv

• Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings: The study examines how large language models represent relationships between languages, finding that similarities between input embeddings are highly interpretable and that the geometry of these embeddings differs between model families. arXiv

• Enhancing Interpretability Using Human Similarity Judgements to Prune Word Embeddings: This research presents a supervised-learning method that identifies subsets of model features in word embeddings that improve prediction of human similarity judgments, enhancing interpretability in specific semantic domains. arXiv

• Interpretable Neural Embeddings with Sparse Self-Representation: The paper proposes a method to learn expressive, interpretable word embeddings by associating data self-representation with a shallow neural network, achieving comparable performance to baseline embeddings while enhancing interpretability. arXiv

• Word Equations: Inherently Interpretable Sparse Word Embeddings through Sparse Coding: This work introduces a method to transform pre-trained dense word embeddings into sparse, interpretable embeddings, where each dimension corresponds to a natural language word or specific grammatical concept. arXiv

• Imparting Interpretability to Word Embeddings While Preserving Semantic Structure: The study introduces an additive modification to the objective function of embedding learning algorithms to align words along predefined concepts, imparting interpretability to word embeddings by assigning meaning to vector dimensions. Cambridge University Press & Assessment

• Decoding Word Embeddings with Brain-Based Semantic Features: This research examines the interpretability of word embeddings by aligning them with semantic features derived from brain activity, aiming to understand the semantic information encoded within embedding dimensions. MIT Press Direct

• Semantic Structure and Interpretability of Word Embeddings: The paper investigates the semantic structure of word embeddings and proposes methods to enhance their interpretability, addressing the challenge of understanding the meaning encoded in embedding dimensions. IEEE Xplore

• From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions for Large Language Models: This comprehensive review traces the evolution from word embeddings to multimodal embeddings, discussing techniques, applications, and future directions in the context of large language models. arXiv

Distributional Semantics

• Investigating the Extent to which Distributional Semantic Models Capture a Broad Range of Semantic Relations: This study evaluates how well various distributional semantic models (DSMs) capture a wide array of semantic relations, providing insights into their effectiveness and limitations. Home | Aurora

• Distributional Semantics and Linguistic Theory: This paper explores the intersection of distributional semantics and linguistic theory, discussing how distributional methods can inform and be informed by linguistic insights. arXiv

• Distributional Semantics, Holism, and the Instability of Meaning: The authors examine the holistic nature of distributional semantics and address concerns regarding the stability and variability of meaning within distributional frameworks. arXiv

• Distributional Social Semantics: Inferring Word Meanings from Social Network Structure: This research introduces a model that infers word meanings based on social network structures, integrating social and distributional semantics. BT Johns

• A Comparative Evaluation and Analysis of Three Generations of Distributional Semantic Models: The paper provides a comparative evaluation of different generations of DSMs, analyzing their performance across various semantic tasks. arXiv

• Distributional Semantics: Meaning Through Culture and Interaction: This article discusses how distributional semantics can be enriched by incorporating cultural and interactional contexts, emphasizing the role of social factors in meaning construction. BPB

• Chapter 5: Distributional Theories of Meaning: Experimental Approaches: This chapter reviews experimental approaches to distributional theories of meaning, highlighting methodologies and findings in the field. Springer Link

• Divergent Semantic Integration (DSI): Extracting Creativity from Semantic Networks: The study introduces the concept of Divergent Semantic Integration, proposing a method to extract creative insights from semantic networks using distributional principles. Springer Link

• Using Distributional Principles for the Semantic Study of Contextual Phenomena: This paper applies distributional principles to analyze contextual semantic phenomena, demonstrating the utility of distributional methods in understanding language in context. ACL Anthology

• Formal Distributional Semantics: Introduction to the Special Issue: An introduction to a special issue on formal distributional semantics, discussing the integration of formal semantic theories with distributional approaches. ACL Anthology

Word Analogy Tasks

• Contrastive Loss is All You Need to Recover Analogies as Parallel Lines: This paper demonstrates that employing a contrastive loss function over distributional information enables the recovery of analogies as parallel lines in high-dimensional space, offering a competitive and efficient alternative to traditional word embedding models.

• Predicting Drug-Gene Relations via Analogy Tasks with Word Embeddings: The study explores the application of word embeddings to predict drug-gene relationships through analogy tasks, highlighting the potential of embedding models in biomedical informatics.

• Analogies minus Analogy Test: Measuring Regularities in Word Embeddings: This research decomposes and analyzes the classic arithmetic word analogy test, introducing new metrics to assess the presence of linguistic regularities in word embeddings beyond traditional analogy evaluations.

• Enhancing Semantic Word Representations by Embedding Deeper Word Relationships: The paper proposes a method to enrich word embeddings by incorporating deeper semantic relationships, improving the embeddings' ability to capture complex word analogies and relationships.

• Word Embedding for Analogy Making: This study addresses the limitations of current word embeddings in analogy tasks and introduces an approach to enhance their capability in capturing conceptual relationships for improved analogy making.

• The Word Analogy Testing Caveat: The paper discusses potential pitfalls in word analogy testing, emphasizing the impact of normalization and vector space distortions on the evaluation of word embeddings.

• Analogies Explained: Towards Understanding Word Embeddings: This work provides a probabilistic explanation for the linear relationships observed in word embeddings, offering insights into why certain word analogies form parallelograms in embedding spaces.

• Sentence Analogies: Linguistic Regularities in Sentence Embeddings: The study extends the concept of word analogies to sentence embeddings, investigating whether sentence-level embeddings capture analogous relationships similar to word-level embeddings.

• Evaluating Word Embeddings Using a Representative Suite of Practical Tasks: This research proposes a standardized suite of downstream tasks to evaluate word embeddings, moving beyond intrinsic evaluations like word similarity and analogy tasks to assess practical applicability.

• Visual Exploration of Semantic Relationships in Neural Word Embeddings: The paper presents visualization techniques to explore semantic relationships in neural word embeddings, aiding in the understanding and interpretation of embedding spaces concerning analogy tasks.

• Unsupervised Learning of Distributional Relation Vectors: Introduces a novel unsupervised method to directly learn relation vectors from word co-occurrence statistics, offering an alternative to approaches that manipulate pre-trained word embeddings.

Embedding Geometry

• The Geometry of Culture: Analyzing Meaning through Word Embeddings: This study demonstrates how neural-network word embedding models can be utilized for large-scale text analysis, revealing that these models produce richer insights into cultural associations and categories than prior methods. arXiv

• Grassmannian Learning: Embedding Geometry Awareness in Machine Learning: This paper introduces Grassmannian learning, which embeds geometry awareness into machine learning models, enhancing their ability to handle subspace-structured features and orthogonality constraints. arXiv

• On the Dimensionality of Word Embedding: The authors investigate the unitary-invariance property of word embeddings and discuss how most existing algorithms can be formulated as low-rank matrix approximations, either explicitly or implicitly. NeurIPS Proceedings

• Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric Methods: This guide explores how modern machine learning increasingly encounters richly structured data that is inherently non-Euclidean, necessitating a broader mathematical perspective to extract knowledge from such data. arXiv

• Dissecting Embedding Methods: Learning Higher-Order Geometries from Data: The paper delves into the theory of manifold learning and finding lower-dimensional manifold representations, addressing issues related to the curse of dimensionality in geometric deep learning methods. arXiv

• Locally Linear Embedding in Machine Learning: This article explains how Locally Linear Embedding (LLE) is an unsupervised approach designed to transform data from its original high-dimensional space into a lower-dimensional space, capturing the inherent geometry of nonlinear manifolds. GeeksforGeeks

• Word Embeddings: Explaining Their Properties: This work provides a theoretical understanding of word embeddings, capturing the meaning of a word using a low-dimensional vector, and discusses their fascinating properties in natural language processing. OffConvex

• Word Embeddings in NLP: The article discusses how word embeddings represent words in a lower-dimensional space, allowing words with similar meanings to have similar representations, and their role in extracting features from text for machine learning models. GeeksforGeeks

• What Are Embeddings in Machine Learning?: This piece explains how embeddings transform high-dimensional data into a lower-dimensional space while preserving essential relationships and properties, playing a crucial role in various machine learning tasks. GeeksforGeeks

• Word Embeddings - GitHub Pages: The tutorial provides insights into word embedding models, discussing their training objectives, properties, and the geometric relationships they capture in the embedding space. Lena Voita

Morphological Transformations

• Unsupervised Morphology Induction Using Word Embeddings: This paper presents a language-agnostic, unsupervised method for inducing morphological transformations between words using word embeddings, capable of discovering a wide range of morphological transformations and providing high-quality embeddings for out-of-vocabulary and low-count morphological variants.

• Morphological Word-Embeddings: The study introduces a multi-task objective that jointly predicts the next word and its morphological tag given a history, aiming to improve word embeddings by incorporating morphological information.

• Morphological Skip-Gram: Using Morphological Knowledge to Improve Word Representation: This research proposes a method that replaces the FastText bag of character n-grams with a bag of word morphemes through morphological analysis, enhancing word embeddings by capturing the inner structure of words.

• MorphTE: Injecting Morphology in Tensorized Embeddings: The paper introduces MorphTE, a word embedding compression method with morphological augmentation, representing a word embedding as an entangled form of its morpheme vectors via the tensor product, which injects prior semantic and grammatical knowledge into the learning of embeddings.

• Derivational Morphological Relations in Word Embeddings: This study evaluates the ability of word embeddings to identify properties of word derivations in the morphologically rich Czech language, analyzing derivational relations between pairs of words.

Cross-Lingual Embedding Alignment

• Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing: This paper introduces a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion, constructing context-independent variants of the original monolingual spaces to derive an alignment for the context-dependent spaces.

• Hierarchical Mapping for Crosslingual Word Embedding Alignment: The study proposes a hierarchical mapping approach for aligning word embedding spaces in different languages into a common crosslingual space, addressing biases towards the choice of the pivot language and considering language proximity and linguistic typology.

• Improving Cross-Lingual Word Embeddings by Meeting in the Middle: This research shows that cross-lingual word embeddings can be effectively learned by aligning two disjoint monolingual vector spaces through linear transformations, using no more than a small bilingual dictionary as supervision.

• Cross-Lingual Alignments of ELMo Contextual Embeddings: The paper addresses the challenge of producing cross-lingual mappings of contextual embeddings by proposing several cross-lingual mapping methods for ELMo embeddings, including linear mapping methods and novel nonlinear ELMoGAN mapping methods based on GANs.

• A Survey of Embedding Space Alignment Methods for Language and Vision: This survey considers word-to-word, word-to-sentence, and sentence-to-sentence alignment tasks as solutions to embedding space alignment, extending past basic linear maps into the realm of deep learning models.

Subspace Identification

• Dissecting Embedding Methods: Learning Higher-Order Geometries from Data: The paper delves into the theory of manifold learning and finding lower-dimensional manifold representations, addressing issues related to the curse of dimensionality in geometric deep learning methods.

• Grassmannian Learning: Embedding Geometry Awareness in Machine Learning: This paper introduces Grassmannian learning, which embeds geometry awareness into machine learning models, enhancing their ability to handle subspace-structured features and orthogonality constraints.

• Locally Linear Embedding in Machine Learning: This article explains how Locally Linear Embedding (LLE) is an unsupervised approach designed to transform data from its original high-dimensional space into a lower-dimensional space, capturing the inherent geometry of nonlinear manifolds.

• Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric Methods: This guide explores how modern machine learning increasingly encounters richly structured data that is inherently non-Euclidean, necessitating a broader mathematical perspective to extract knowledge from such data.

• On the Dimensionality of Word Embedding: The authors investigate the unitary-invariance property of word embeddings and discuss how most existing algorithms can be formulated as low-rank matrix approximations, either explicitly or implicitly.

Probing Classifiers

• Probing Classifiers: Promises, Shortcomings, and Advances: This article critically reviews the use of probing classifiers in interpreting deep neural network models for NLP, discussing their benefits, limitations, and recent methodological improvements. MIT Press Direct

• Probing as Quantifying Inductive Bias: The paper proposes a Bayesian framework to measure the inductive biases encoded in pretrained contextual representations, addressing challenges in traditional probing methodologies. arXiv

• Probing BERT in Hyperbolic Spaces: This work explores the effectiveness of hyperbolic spaces in probing tasks, aiming to uncover hierarchical linguistic structures within BERT's embeddings. arXiv

Contextual Embeddings (BERT, GPT)

• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings: This study examines the extent to which models like BERT, ELMo, and GPT-2 produce context-specific representations, analyzing their geometric properties. arXiv

• Contextual Word Representations with BERT and Other Pre-trained Language Models: A presentation detailing the architecture and applications of BERT and similar models, highlighting their impact on NLP tasks. Stanford University

• Rethinking of BERT Sentence Embedding for Text Classification: This research revisits the use of BERT for sentence embeddings in text classification, proposing enhancements for better performance. Springer Link

Bias and Fairness in NLP

• Assessing Social and Intersectional Biases in Contextualized Word Representations: The paper analyzes biases related to gender, race, and intersectional identities in models like BERT and GPT-2, proposing methods to assess and mitigate such biases. arXiv

• Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads: This study investigates how biases manifest within the attention mechanisms of transformer-based models, offering insights into their internal bias behaviors. arXiv

• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders: The authors introduce FairFil, a method to reduce biases in pretrained text encoders without requiring retraining, enhancing fairness in NLP applications. arXiv

Manifold Learning in Representation Spaces

• Probing BERT in Hyperbolic Spaces: This work explores the effectiveness of hyperbolic spaces in probing tasks, aiming to uncover hierarchical linguistic structures within BERT's embeddings. arXiv

• Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric Methods: This guide explores how modern machine learning increasingly encounters richly structured data that is inherently non-Euclidean, necessitating a broader mathematical perspective to extract knowledge from such data.

• Dissecting Embedding Methods: Learning Higher-Order Geometries from Data: The paper delves into the theory of manifold learning and finding lower-dimensional manifold representations, addressing issues related to the curse of dimensionality in geometric deep learning methods.

[thorwhalen]

relational vector mining

Moved to this discussion

[thorwhalen]

Nearly Orthogonal Directions in High Dimensions

Working on code for this here

In an ( n )-dimensional Euclidean space ( \mathbb{R}^n ), we can have at most ( n ) mutually orthogonal unit vectors. However, as ( n ) increases, we can fit exponentially many nearly orthogonal vectors—vectors whose pairwise dot products are small but nonzero. This is a well-known phenomenon in high-dimensional geometry.

1. Expected Dot Product Between Random Unit Vectors

If we pick two random unit vectors ( \mathbf{v}_1, \mathbf{v}_2 ) from the unit sphere ( S^{n-1} ), their dot product ( \cos \theta ) follows a normal distribution with mean zero and variance:

As n --> infinity, almost all vectors become nearly orthogonal.

Concentration of Measure in High Dimensions

2. Distribution of Dot Products in High Dimensions

For two random unit vectors, the probability density function (PDF) of their dot product ( x = \cos \theta ) is approximately:

This shows that as ( n ) grows, the dot products cluster around zero (i.e., angles near ( 90^\circ )).

See Distribution of scalar products of two random unit vectors in 𝐷 dimensions

3. Number of Nearly Orthogonal Vectors

This comes from the Johnson-Lindenstrauss Lemma

If we define near-orthogonality as:

for some small epsilon, the maximum number of such vectors satisfies:

This means that the number of nearly orthogonal vectors grows exponentially with dimension ( n ), making high-dimensional spaces highly expressive.

See:
High Dimensional Geometry, the Johnson-Lindenstrauss Lemma, MinHash and
High Dimensional Geometry, Curse of Dimensionality, Dimension Reduction

Also this video.

4. Applications

• Machine Learning: High-dimensional spaces allow large-margin classifiers (e.g., SVMs) to work well by increasing separation between classes.
• Error-Correcting Codes: Large sets of nearly orthogonal vectors are useful for designing robust codes.
• Neural Networks: Feature representations in deep learning often leverage high-dimensional embeddings to minimize redundancy.

As dimensions increase, the behavior of nearly orthogonal vectors becomes an essential property in many applications!