Un-supervised: Crisis Tweet Clustering

An unsupervised learning pipeline for crisis tweet analysis using the HumAID and CrisisBench datasets. The project focuses on discovering latent structure in humanitarian crisis tweets via preprocessing, sentence embeddings, dimensionality reduction, and clustering.

---

Overview

This repository implements an end-to-end unsupervised NLP workflow for humanitarian crisis tweets. The pipeline is modular and notebook-driven, starting from raw text and ending with interpretable tweet clusters suitable for downstream analysis or prototyping supervised models.

Key components:

• Text preprocessing and normalization
• Exploratory data analysis (EDA)
• Sentence-level feature extraction
• Dimensionality reduction (UMAP, PCA, t-SNE)
• KMeans clustering with elbow and silhouette-based evaluation

---

Pipeline

Notebook	Description
01_data_download.ipynb	Download HumAID and CrisisBench datasets
02_preprocessing.ipynb	Clean and normalize tweet text (URLs, mentions, emojis, casing, etc.)
03_eda.ipynb	Exploratory analysis: label distributions, tweet length, hashtags
04_feature_extraction.ipynb	Generate sentence embeddings for tweets
05_dimensionality_reduction.ipynb	Compare UMAP, PCA, t-SNE and run small grid searches
06_clustering.ipynb	KMeans clustering with elbow method and silhouette analysis

Run the notebooks in order (01 -> 06) to reproduce the full pipeline.

---

Project Structure

Un-supervised/
├── notebooks/          # Jupyter notebooks (pipeline steps)
├── config/             # YAML configuration files
├── data/
│   └── processed/      # Cleaned data, embeddings, intermediate artifacts
├── figures/            # Generated plots and visualizations
├── scripts/            # Shell scripts to run notebooks end-to-end
├── requirements.txt    # Python dependencies
└── plan.md             # Project plan and design notes

---

Setup

1. Clone the repository

git clone https://github.com/Cyril-36/Un-supervised.git
cd Un-supervised

2. Create and activate a virtual environment

python -m venv .venv
source .venv/bin/activate        # Linux / macOS
# .venv\Scripts\activate         # Windows

3. Install dependencies

pip install -r requirements.txt

---

How to Run

Option 1: End-to-end via script

bash scripts/run_notebooks.sh

This executes all pipeline steps sequentially and populates data/processed and figures.

Option 2: Step-by-step in Jupyter

1. Launch Jupyter:

jupyter notebook

2. Open the notebooks in notebooks/ from 01_data_download.ipynb through 06_clustering.ipynb and run them in order.

---

Datasets

• HumAID - Humanitarian AI dataset of crisis tweets with event-level annotations.
• CrisisBench - Multi-event crisis tweet benchmark including events such as:
  • Alberta floods
  • Nepal earthquake
  • Hurricane Harvey

Refer to the original dataset documentation for licensing, terms of use, and citation requirements.

---

Technical Stack

---

Possible Extensions

• Experiment with alternative embedding models (e.g., different sentence-transformers variants).
• Evaluate additional clustering algorithms (HDBSCAN, spectral clustering, Gaussian mixtures).
• Add automatic cluster labeling and qualitative analysis reports or dashboards.
• Extend the pipeline to multilingual crisis tweet streams.

---

License

This project is intended to be released under the MIT License. Add a LICENSE file in the repository root to finalize the licensing information.