Latent Dirichlet Allocation

Latent Dirichlet Allocation

Overview

Latent Dirichlet Allocation (LDA) is used in CRISP-T for topic modeling to extract latent themes from textual corpora. Topics can be assigned to documents and utilized for grounded theory coding, selective sampling, and triangulation with numeric analyses.

Implementation

• --topics: Builds an LDA model; --assign: assigns topics/keywords to documents.
• crispviz --ldavis: Generates an interactive topic visualization (pyLDAvis).
• Topic number defaults to 8 (following guidance noted in repository documentation), configurable via --num or --topics-num in crispviz.

Workflow

1. Import corpus and run --topics then --assign.
2. Inspect document metadata for assigned keywords and topic distributions.
3. Use Semantic filtering to retrieve documents by keywords (e.g., --filters keywords=mask).
4. Compare topic prevalence with quantitative patterns (e.g., regression coefficients).

Sense-making

• Align topics with high-importance features from decision trees or strong coefficients from regression.
• Use topic-word distributions to refine category names and definitions.
• Sample boundary documents where topics overlap to test category robustness.

Examples

Business

Topic emphasizing "integration" terms (e.g., API, workflow) co-occurs with high training_hours and low api_errors_rate. Narrative excerpts corroborate a category about enablement.

Medical

Topic featuring "sleep," "fatigue," "energy" aligns with low sleep_balance and high fatigue_score, supporting a theory of sleep-mediated fatigue.

References

• Mettler et al. (2025) — Methodological reflection on computational text analysis.

See Also

• NLP based techniques
• Summarization techniques
• Semantic filtering
• Human Sense-making and triangulation