Sludge Classifiers: Code and Data

This repository contains the code and annotated data accompanying the paper:

Chesterfield, A., Gillespie, A., Goddard, A. and Krpan, D. (2026). Feeling the Friction: Developing and validating text classifiers for sludge in consumer complaints. (manuscript in preperation)

It supports two classifiers for detecting sludge in consumer financial complaints:

• Process Sludge: complaints describing barriers when trying to do something or get help doing it (e.g., procedural steps and human interactions needed to complete tasks and achieve goals)
• Informational Sludge: complaints describing barriers when trying to find, access, or understand key information about a product, service, or process

Both classifiers were evaluated using:

1. RoBERTa-base fine-tuned on 825 annotated complaints
2. Few-shot LLM classification using Claude Sonnet 4.5 and GPT-5.2

---

Repository Structure

Sludge_Classifiers_Public/
├── data/
│   └── cfpb_sludge_annotated_971.csv       # Annotated CFPB complaints (n=971)
├── roberta_classifier/
│   ├── run_process_sludge_final_test.py    # Train & evaluate on held-out test set
│   ├── run_informational_sludge_final_test.py
│   ├── process_sludge_grid_search.ipynb    # Hyperparameter search notebook
│   ├── informational_sludge_grid_search.ipynb
│   ├── compute_bootstrap_ci.py             # Bootstrap CIs for RoBERTa results
│   ├── splits/                             # Pre-computed train/test/fold indices
│   │   ├── process_train_pool_ids.json
│   │   ├── process_test_ids.json
│   │   ├── process_fold_indices.json
│   │   ├── informational_train_pool_ids.json
│   │   ├── informational_test_ids.json
│   │   └── informational_fold_indices.json
│   └── results/
│       ├── process_sludge_test_results.json
│       └── informational_sludge_test_results.json
└── llm_classifier/
    ├── llm_final_evaluation.py             # Run LLM few-shot classification
    ├── calculate_metrics.py                # Compute metrics from LLM predictions
    ├── calculate_irr.py                    # Inter-rater reliability (Claude vs GPT)
    ├── compute_bootstrap_ci.py             # Bootstrap CIs for LLM results
    ├── prompts/
    │   ├── process_sludge_prompt.txt
    │   └── informational_sludge_prompt.txt
    └── results/
        ├── llm_final_metrics.json          # Pre-computed LLM results
        ├── process_sludge_predictions.csv  # Per-complaint predictions
        ├── informational_sludge_predictions.csv
        └── bootstrap_confidence_intervals.json

---

Dataset

data/cfpb_sludge_annotated_971.csv contains 971 deduplicated CFPB consumer complaints (debt collection category) with manual sludge annotations.

Columns:

Column	Description
Consumer complaint narrative	Full complaint text
Complaint ID	CFPB complaint identifier
Sludge yes (1) or no (0)	Any sludge present (either type)
Process sludge (1 or 0)	Process sludge label (31.9% positive)
Process sludge (text excerpt)	Supporting text excerpt from coder
Informational (1 or 0)	Informational sludge label (16.2% positive)
Informational (text excerpt)	Supporting text excerpt from coder
process_split	train or test for process sludge classifier
informational_split	train or test for informational sludge classifier

Note on splits: Each classifier uses its own independently stratified 85/15 split (825 train, 146 test). The two test sets are not identical — they were stratified on their own label to ensure representative class proportions in each held-out set.

---

Pre-computed Results

If you just want the reported results, see:

• roberta_classifier/results/ — RoBERTa final test metrics and confusion matrices
• llm_classifier/results/llm_final_metrics.json — LLM point estimates for all models
• llm_classifier/results/bootstrap_confidence_intervals.json — 95% bootstrap CIs for LLM models
• Run roberta_classifier/compute_bootstrap_ci.py to regenerate RoBERTa bootstrap CIs

Summary of Results (MCC, 95% CI)

Task	RoBERTa-base	GPT-5.2	Claude Sonnet 4.5
Process Sludge	0.795 [0.693, 0.882]	0.796 [0.698, 0.889]	0.766 [0.648, 0.874]
Informational Sludge	0.621 [0.433, 0.789]	0.706 [0.552, 0.850]	0.546 [0.433, 0.654]

All CIs overlap across models; observed differences are within chance at this sample size.

Model weights (Hugging Face)

The fine-tuned RoBERTa classifiers are published on Hugging Face and can be loaded directly without retraining:

• AlexChesterfield/sludge-process-roberta
• AlexChesterfield/sludge-informational-roberta

---

Re-running the RoBERTa Classifier

Requirements

pip install torch transformers datasets scikit-learn pandas numpy

Tested with Python 3.10+, PyTorch 2.x. Uses MPS (Apple Silicon GPU) if available, otherwise CPU.

Run the final evaluation

Run from the repository root (Sludge_Classifiers_Public/):

# Process sludge
python roberta_classifier/run_process_sludge_final_test.py

# Informational sludge
python roberta_classifier/run_informational_sludge_final_test.py

Each script:

1. Trains RoBERTa-base on the 825-sample train pool (balanced class weights, 4 epochs, lr=2e-5)
2. Evaluates once on the 146-sample held-out test set
3. Saves results and model weights to results/

To prevent your Mac from sleeping during training:

caffeinate -dims python roberta_classifier/run_process_sludge_final_test.py

Re-run the hyperparameter search (optional)

Open the grid search notebooks in Jupyter:

jupyter notebook roberta_classifier/process_sludge_grid_search.ipynb
jupyter notebook roberta_classifier/informational_sludge_grid_search.ipynb

These notebooks search over learning rate, batch size, epochs, and weight decay using cross-validation on the train pool only. The test set is never loaded during this phase.

---

Re-running the LLM Classifier

Requirements

pip install anthropic openai pandas python-dotenv

Set your API keys:

export ANTHROPIC_API_KEY="your-key-here"
export OPENAI_API_KEY="your-key-here"

Or create a .env file in llm_classifier/:

ANTHROPIC_API_KEY=your-key-here
OPENAI_API_KEY=your-key-here

Run the evaluation

Run from the llm_classifier/ directory:

cd llm_classifier

# Both models, both classifiers (~584 API calls total)
python llm_final_evaluation.py --model both --classifier both

# Single model/classifier
python llm_final_evaluation.py --model claude --classifier process
python llm_final_evaluation.py --model gpt52 --classifier informational

Predictions are saved to results/ as {model}_{classifier}_predictions.json.

Compute metrics and CIs

# Compute classification metrics (MCC, F1, Precision, Recall)
python calculate_metrics.py

# Compute inter-rater reliability between Claude and GPT
python calculate_irr.py

# Compute bootstrap CIs (run roberta bootstrap CI script first)
cd ../roberta_classifier && python compute_bootstrap_ci.py
cd ../llm_classifier && python compute_bootstrap_ci.py

---

Applying to Your Own Data

To classify your own CFPB complaints (or similar consumer complaint text):

Option A — LLM (no training required):

1. Format your complaints as a CSV with a text column
2. Modify llm_classifier/llm_final_evaluation.py to load your CSV
3. The prompts in llm_classifier/prompts/ can be used as-is

Option B — RoBERTa (fine-tuned, better performance):

1. Load the published model weights directly from Hugging Face — no training required:

from transformers import pipeline

# Process sludge classifier
process_clf = pipeline("text-classification",
                       model="AlexChesterfield/sludge-process-roberta")

# Informational sludge classifier
info_clf = pipeline("text-classification",
                    model="AlexChesterfield/sludge-informational-roberta")

complaint = "I called five times and was transferred each time with no resolution."
print(process_clf(complaint))   # LABEL_1 = sludge present, LABEL_0 = absent

2. To retrain from scratch on your own annotated data, use roberta_classifier/run_process_sludge_final_test.py (modify paths accordingly).

---

Codebook

The annotation codebook defining process and informational sludge is provided in the supplementary materials of the paper. Brief definitions:

• Process Sludge: Barriers faced when trying to DO something or GET HELP doing it. This includes both procedural steps AND human interactions needed to complete tasks and achieve goals.
• Informational Sludge: Barriers when trying to FIND, ACCESS or UNDERSTAND key information about the product, service, or process.

---

Citation

If you use this code or data, please cite:

Chesterfield, A., Gillespie, A., Goddard, A. and Krpan, D. (2026). Feeling the Friction:
Developing and validating text classifiers for sludge in consumer complaints.
Manuscript under review. Journal and DOI to be added on publication.

---

Contact

Alexandra Chesterfield Department of Psychological and Behavioural Science London School of Economics and Political Science a.m.chesterfield@lse.ac.uk