Skip to content
003miles edited this page May 1, 2025 · 6 revisions

PressScan

A local, high-throughput LLM-based sentiment analysis tool for structured text data (e.g. newspaper articles).

Purpose

PressScan is a command-line Python tool for high-volume sentiment classification of .csv or .xlsx text datasets using a local large language model (LLM). It was created to support academic and exploratory research, especially in domains where traditional sentiment analysis methods fail to capture nuance, such as journalism, political discourse, or reporting on migration.

Prerequisites

Before you begin, ensure you have the following installed:

Why are these needed?

  • Ollama manages local LLM serving (e.g., Mistral, LLaMA 3, Deepseek, etc.)
  • Python 3.9+ ensures compatibility with libraries like pandas

Setup Instructions

1. Start the Ollama Server

ollama serve

2. Pull the Required Model

ollama pull mistral

To use a different model:

ollama pull llama3

Then pass it later using --model llama3

3. Clone the Repository

git clone https://github.com/003miles/PressScan.git
cd PressScan

4. Install Required Packages

pip install -r requirements.txt

Recommended to use a virtual environment (e.g., python -m venv .venv)

5. Run the Sentiment Analyser

python cli_sentiment.py --file <CSV_FILE_PATH> [OPTIONS]

Example:

python cli_sentiment.py --file dataset.csv --model mistral --columns title,quotes --choices positive,negative,neutral,unrelated --sample 1000 --workers 8 --debug

Command Line Options

Option Description
--file (Required) Path to the input CSV file
--model Model name to use (default: mistral)
--columns Comma-separated list of columns to combine for text input
--prompt Custom prompt text file (default: prompt.txt)
--choices Comma-separated list of classification labels
--sample Sample size (default: 1000 rows)
--workers Max workers for CPU threading (default: 4)
--debug Output full LLM reasoning text along with answers
--full Ignore sampling and run on the entire dataset
--dryrun Process without saving the output CSV
You can see help any time:
python cli_sentiment.py --help

Output

The final annotated CSV file will be saved in the temp_results/ folder.

Example:

temp_results/annotated_dataset.csv_899f4b62-0383-4f87-9d9d-e86b2249186c.csv

Example Output

title quotes Sentiment
Muslim girl denied entry at McDonald's Security guard told her to remove hijab Neutral
Migrants accused of abusing welfare Minister claims abuse is "out of control" Negative
Migrants praised for supporting NHS Article celebrates healthcare workers Positive
Labour donor criticises healthcare No mention of migrants Unrelated

How Prompting Works

You provide only the classification logic in prompt.txt.

The script dynamically constructs the full prompt like this:

Text: {row_text}
Instructions: {user_prompt}
Classification choices: {choices}

YOU MUST RESPOND FOLLOWING THE FORMAT BELOW
Reasoning:
Answer:

Why Require "Reasoning"?

Instructing the LLM to first explain its reasoning greatly improves reliability. Instead of immediately guessing a label, the model "thinks aloud" before answering. This technique is based on chain-of-thought prompting (Wei et al., 2022), which boosts model accuracy on complex tasks.

Enabling a structured "Reasoning:" field forces more deliberate, contextual classifications. It also allows manual review, aiding prompt refinement (detailed in the next section)

Prompt Refinement

Prompt wording strongly influences results. To refine your setup:

  • Start with small samples (--sample 100) and inspect outputs
  • Adjust your prompt.txt for clarity and label definitions
  • Prefer explicit instructions over vague sentiment terms Debugging with raw model output (see Debug Mode) can highlight where confusion arises

Debug Mode

Enable debug mode with --debug:

  • Outputs the full text of each LLM response (reasoning + answer)
  • Useful to diagnose misclassifications or improve prompts
  • Included in addition to extracted sentiment column Example CLI:
python cli_sentiment.py --file articles.csv --debug

Typical debug output excerpt:

Reasoning: The article discusses hate crimes faced by Muslim students after a terror attack. It portrays Muslims as victims of discrimination.
Answer: Neutral

Running Locally with Ollama

The tool runs LLMs via Ollama, which uses llama.cpp and GGUF model files.

Why Mistral?

  • Reasoning performance is strong for its size (7B parameters)
  • Available in 4-bit quantised versions for efficient local inference

Hardware Suggestions

Component Recommended
CPU 8-16 cores
GPU 12-24 GB VRAM (optional)
RAM 16-32 GB

Ollama automatically manages GPU or CPU usage.

Using RunPod for Cloud Execution (Recommended for Large Datasets)

If your local machine lacks the GPU power or memory needed for large-scale sentiment analysis, RunPod.io provides an affordable and flexible alternative. This section walks through a full example setup based on a tested configuration.

Deployment Steps (Using Default PyTorch Template)

1. Create SSH Key Pair (if you haven't already)

ssh-keygen -t ed25519 -C "your_email@example.com"
cat ~/.ssh/id_ed25519.pub
  • Copy the public key and add it to your RunPod account under Settings > SSH Keys

2. Deploy a Pod

  • Go to RunPod.io
  • Select Deploy > Secure Cloud GPU Pod
  • Choose a GPU (e.g., RTX 5090, 32 GB VRAM)
  • Select the "RunPod PyTorch 2.1" template (includes Python preinstalled)
  • Under Overrides:
    • Choose 200 GB Container Disk (skip volume storage)
  • Enable SSH Terminal Access
  • Confirm and Deploy

3. Connect to the Pod via SSH

ssh root@<your_runpod_ip> -p <runpod_port> -i ~/.ssh/id_ed25519

The Connect button, under your deployed pod, should give you an automatically generated SSH connection command

4. Accessing Output Files via SFTP

I recommend the FileZilla client to access the container disk where generated output will be stored. A connection can be made using the same IP, Port and SSH key used to connect. A guide on setting this up can be found here

Environment Setup on the Pod

You should first install lshw so that Ollama can access the GPU, as per RunPod's official documentation. I also recommend tmux to efficiently manage your workflow with panes or windows, all within a single terminal instance. Run the following commands on your pod:

apt update
apt install lshw tmux
tmux new

Once tmux is running, use tmux new-window to create your windows. tmux set mouse enables you to select windows using the mouse.

1. Start Ollama and Download a Model

curl -fsSL https://ollama.com/install.sh | sh
ollama serve

And in a new window:

ollama pull mistral

2. Clone the PressScan Repository

git clone https://github.com/003miles/PressScan.git
cd PressScan

3. Create a Virtual Environment and Install Dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

4. Upload your Datasets

Using either an SFTP client (like FileZilla) or scp, copy your datasets to the PressScan directory. You can use ls in the remote terminal to verify the files were transferred.

5. Run the Analyser

python cli_sentiment.py --file dataset.csv --columns title,quotes \
    --choices positive,negative,neutral,unrelated \
    --model mistral --sample 1000 --workers 16 --debug

Limitations

  • Output quality depends heavily on clear prompts
  • LLMs occasionally misinterpret sarcasm or irony

Theoretical Foundations and Rationale

PressScan uses large language models (LLMs) to infer sentiment in structured text (e.g. news articles) without requiring prior domain-specific training. This approach is informed by recent research in instruction-based prompting, zero-shot classification and chain-of-thought reasoning.

Why Use LLMs for Sentiment Analysis?

Traditional sentiment analysis methods, such as lexicon-based systems, logistic regression, support vector machines or pre-trained classifiers, rely heavily on domain-specific labeled data, which is often incredibly difficult to obtain. While effective in constrained contexts (e.g. product reviews), these methods struggle in domains where sentiment is nuanced, implicit, or highly contextual, such as:

  • Political reporting and ideological framing
  • Public discourse on polarising social issues

Large language models, by contrast:

  • Are pre-trained on diverse web-scale corpora, capturing a wide range of human language patterns and discursive strategies
  • Can generalise to novel sentiment tasks via carefully constructed prompts (Brown et al., 2020)
  • Show emerging capabilities for reasoned judgement when structured to "think aloud" (Wei et al., 2022)

This makes LLMs particularly suitable for analysing sentiment as it is framed by the author or source text, rather than based on superficial lexical cues.

Importantly, the LLM is not asked for its own opinion, but rather to infer the framing or sentiment implied by the author

Prompt Design Philosophy

To encourage more consistent and interpretable outputs, PressScan uses a fixed template that includes:

  • A user-defined instruction (prompt.txt)
  • A list of allowable classification labels
  • A required format for LLMs to follow"
     Reasoning:
     Answer:
    

This structure is based on chain-of-thought prompting (Wei et al., 2022), which has been shown to:

  • Improve model performance on reasoning and classification tasks
  • Reduce ambiguity by forcing the model to justify its output
  • Produce outputs that are easier for humans to inspect, critique, or override

This reasoning-first format is enforced automatically. Users only need to supply their instruction and label choices.

Reproducibility and Annotation Transparency

Several features are designed with research reproducibility in mind:

  • The --sample flag uses a fixed random seed for consistent subsets
  • --debug exposes full model responses (including rationale), allowing users to audit, refine prompts, or conduct secondary analysis
  • The structured output format enables downstream review or post-hoc evaluation This makes PressScan especially useful for qualitative content analysis, exploratory annotation, or as a precursor to manual coding.

Ethical and Interpretive Considerations

LLMs are trained on human data and may reproduce societal biases, especially when analysing sensitive text. Classifications should not be interpreted as objective truths; they reflect inferred sentiment from the text's framing, not from the LLM itself or the tool author. Researchers using PressScan should remain critical of:

  • Biases introduced by pre-training data (e.g. overrepresentation of Western media sources)
  • Situations where subjective framing, sarcasm, or ambiguity leads to unstable outputs
  • The tendency of LLMs to over-explain or over-generalise

Where appropriate, I encourage triangulating results with other qualitative or statistical methods.

Suggested Applications

  • Framing analysis in media studies (e.g. portrayal of migrants, ethnic minorities)
  • Rapid prototyping of annotation schemes before manual coding
  • Comparative studies: LLM vs human judgement
  • Generating candidate labels for weak supervision

References

  • Krugmann, J. & Hartmann, J. (2024). Sentiment Analysis in the Age of Generative AI. Customer Needs and Solutions. https://doi.org/10.1007/s40547-024-00143-4
  • Zhang, W. et al. (2023). Sentiment Analysis in the Era of Large Language Models: A Reality Check. arXiv:2305.15005
  • Nadi, F. et al. (2024). Sentiment Analysis Using Large Language Models: A Case Study of GPT-3.5. In DaSET 2023. https://doi.org/10.1007/978-981-97-0293-0_12
  • Wang, Z. et al. (2024). Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study. arXiv:2304.04339
  • Zou, H. & Wang, Y. (2025). LLM-Augmented Domain Adaptation for Sentiment Analysis. Neurocomputing, 625.
  • Bojić, L. et al. (2025). Comparing LLMs and Human Annotators for Complex Sentiment Tasks. Scientific Reports, 15(1).
  • Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903
  • https://ollama.ai — Local LLM model runner
  • https://github.com/ggerganov/llama.cpp — llama.cpp backend