Cross Modal analysis

Cross Modal Analysis

Cross-Modal Analysis in CRISP-T allows you to bridge the gap between qualitative (textual) and quantitative (numeric) data. By using text metadata fields (such as codes, sentiment labels, or topics) as outcome variables for machine learning tasks, you can validate findings and uncover patterns across different data modalities.

Overview

This feature enables you to:

• Triangulate: Connect qualitative insights with quantitative metrics.
• Validate: Test if numeric variables can predict qualitative codes.
• Integrate: Combine text analysis results seamlessly into ML workflows.

How It Works

The process involves linking text documents to numeric DataFrame rows and then using the metadata from the text (e.g., a "sentiment" tag) as the target for a machine learning model trained on the numeric data (e.g., biological markers).

1. Linkage Methods

To perform cross-modal analysis, your text and numbers must be linked. There are four primary methods:

• ID Linkage (id): Direct matching via explicit IDs (e.g., Patient ID).
• Keyword Linkage (keyword): Matching based on shared tags or keywords.
• Temporal Linkage (temporal): Time-based matching (requires crispt --temporal-link).
• Embedding Linkage (embedding): Semantic similarity matching (requires crispt --embedding-link).

For more details on creating these links, see Data Linking Commands.

2. Aggregation Strategies

Since multiple text documents might link to a single numeric data row, you must specify how to aggregate the text metadata into a single outcome value:

Strategy	Description	Best For
majority	Uses the most common value (vote).	Classification tasks (Categorical outcomes).
mean	Calculates the average.	Regression tasks (Numeric outcomes).
mode	Uses the most frequent value.	Categorical outcomes.
first	Uses the value from the first linked document.	When order is significant.

Running Analysis

You can run cross-modal analysis using the crisp command with ML flags (--regression, --cls, --nnet, etc.).

Basic Syntax

crisp --source <corpus_path> \
      --outcome <text_metadata_field> \
      --linkage <linkage_method> \
      --aggregation <aggregation_strategy> \
      --<ml_task>

Examples

Predicting Sentiment from Numeric Data (Regression)

Predict a numeric sentiment score (from text) based on physiological metrics.

crisp --source ./patient_data \
      --outcome sentiment_score \
      --linkage id \
      --aggregation mean \
      --regression \
      --include "age,bmi,heart_rate"

Classifying Topics based on Demographics (Classification)

Predict which topic a participant discussed based on their demographic data.

crisp --inp ./linked_corpus \
      --outcome topic_label \
      --linkage embedding \
      --aggregation majority \
      --cls \
      --include "income,education,year"

Best Practices

• Data Prep: Ensure your text analysis (e.g., sentiment, topic modeling) is complete and the metadata fields exist before running ML.
• Method Selection: Use majority or mode for categorical labels (classification) and mean for continuous scores (regression).
• Validation: Always check the linkage quality. Low accuracy might mean the links are weak or that the text and numbers capture independent dimensions (which is also a valid finding!).

Troubleshooting

• Error: "No documents with 'X' metadata field": Run the relevant text analysis command first (e.g., crisp --sentiment).
• Error: "No documents linked": Verify your ID matches or run the appropriate crispt --link command before analysis.
• Warning: "Non-numeric values, using majority vote": You tried to use mean on text labels. Switch to majority.