Skip to content

Semantic filtering

Jump to bottom
Bell Eapen edited this page Jan 25, 2026 · 2 revisions

Semantic filtering

Overview

Semantic filtering in CRISP-T aligns textual and numeric data by leveraging shared identifiers, metadata, and similarity-based retrieval. It supports grounded theory by enabling targeted examination of document subsets whose qualitative content can be compared against quantitative patterns (e.g., regression coefficients, cluster assignments).

🔗 1. Linking by shared IDs

If both your text file and your numeric CSV have a column called something like id, CRISP‑T automatically matches them.

Example:

Document 17 → “I felt anxious during the pandemic…”

Row with id = 17 → age = 42, anxiety_score = 8, gender = female

CRISP‑T merges these behind the scenes so that:

When you open Document 17, CRISP‑T knows its numeric attributes

When you filter numeric data, CRISP‑T can show only the matching documents

This is the strongest and most reliable link.

🔍 2. Simultaneous filtering (text + numeric)

If the textual metadata includes keywords matching numeric column names, both datasets are filtered at the same time. For instance, if you have a text dataset of normal and phishing emails labelled with a "spam" topic as yes/no, and a numeric dataset with a "spam" column also marked yes/no, they will be filtered together.

Implementation

# Filter documents by metadata
for key, value in filters.items():
    corpus.documents = [
        d for d in corpus.documents
        if d.metadata.get(key) == value
    ]

# If DataFrame exists, filter it using the same keys
if corpus.dataframe is not None:
    for key, value in filters.items():
        if key in corpus.dataframe.columns:
            corpus.dataframe = corpus.dataframe[corpus.dataframe[key] == value]

This allows you to:

  • Filter documents by numeric themes

  • Explore clusters where text and numbers overlap

  • Build mixed‑methods insights without manually coding everything

CRISP‑T connects them in a graph too!

These keyword links are fuzzy, but they help bridge qualitative and quantitative worlds.

# Create metadata nodes for each numeric column
for col in df.columns:
    graph.add_node(col, type="metadata")

# Connect document → keyword → metadata
for doc in corpus.documents:
    for kw in doc.keywords:
        if kw in df.columns:
            graph.add_edge(doc.id, kw)
            graph.add_edge(kw, col)

Components

Component Mechanism Purpose
Keyword Filters --filters keywords=mask Focus on documents with specific assigned topics/keywords
Cluster Filters --filters cluster=0 Examine thematic concentration within numeric clusters
Semantic Search --semantic QUERY / --semantic-chunks QUERY Retrieve conceptually relevant documents or passages
Similarity Thresholds --rec parameter Control precision vs. recall of semantic retrieval
Cross-modal Alignment Shared IDs Filter numeric DataFrame rows and documents simultaneously

Workflow Example

  1. Assign topics: crisp --inp crisp_input --assign --out crisp_input
  2. Filter by keyword: crisp --inp crisp_input --filters keywords=mask --topics
  3. Perform semantic chunk search for deeper contextual passages: crispt --inp crisp_input --semantic-chunks "public health guidance" --doc-id DOC123 --rec 8.8
  4. Run regression: crisp --inp crisp_input --ml --regression --out crisp_input
  5. Compare documents returned by semantic search against high-magnitude coefficients.

Sense-making and Triangulation

Semantic filtering fosters constant comparison—a core grounded theory technique—by:

  • Enabling retrieval of exemplar documents for emergent categories.
  • Providing chunk-level granularity for coding validation.
  • Supporting theoretical sampling: refine queries to explore underdeveloped conceptual regions (e.g., low-frequency themes).

Business Example

Filter e-commerce reviews by keyword "return" and run semantic search for "policy clarity". Regression shows return_process_time significant. Combined evidence builds a theoretical category around procedural friction affecting satisfaction.

Medical Example

Semantic search for "sleep disturbance" within patient narratives retrieves chunks overlapping with high fatigue_score rows. Logistic regression confirms sleep_balance association—triangulating narrative emphasis and quantitative effect.

Best Practices

  • Use high similarity thresholds (e.g., >0.8) for confirmatory analysis; lower thresholds for exploratory sampling.
  • Document semantic queries and resulting document IDs in analytic memos.
  • Chain filters cautiously to avoid overly narrow subsets that impede theoretical saturation.

Limitations

  • Embedding-based similarity approximates semantic relevance; manual review remains necessary.
  • Over-reliance on single queries risks confirmation bias—vary phrasing.

Future Enhancements

  • Automated suggestion of counterfactual queries to challenge emerging propositions.
  • Integration of semantic chunk tagging directly into corpus metadata for advanced filtering.
  • Multi-query consensus scoring to reduce retrieval variance.

References

  • Mettler et al. (2025) — Computational text analysis for qualitative research.

See Also

Clone this wiki locally