keyword_comparator

A Python tool that compares two text files and identifies keywords significantly overrepresented in one file compared to the other, using Log-likelihood (LL) statistics.

Date

• Initial creation: 2026-01-21
• Uploaded to GitHub: 2026-07-30

Features

• Reads two .txt files
• Cleans text (lowercase, punctuation/digit removal)
• Tokenizes and removes English stopwords
• Computes Log-likelihood (LL) for each word
• Filters words with LL > 3.84 (p < 0.05)
• Excludes words that are not relatively more frequent in target
• Outputs Excel with two sheets:
  • sample1_vs_sample2: sample1 as target, sample2 as reference
  • sample2_vs_sample1: sample2 as target, sample1 as reference
• Each sheet contains: target_corpus, reference_corpus, keyword, target_freq, reference_freq, log_likelihood, p_value

Repository Structure

• keyword_comparator.py – main Python script (run directly)
• keyword_comparator.ipynb – Jupyter Notebook version for interactive execution
• sample1.txt – example input text file 1 (you provide your own)
• sample2.txt – example input text file 2 (you provide your own)
• sample_keyword_comparison.xlsx – example output file (generated after running the script)
• requirements.txt – Python dependencies
• README.md – project documentation (this file)

Usage

Prepare Input Files

• Place two text files as sample1.txt and sample2.txt in the same folder as the script.

Customize (Optional)

• You can adjust the following parameters in the script: -LL_THRESHOLD = 3.84 – significance threshold (p < 0.05) -MIN_TARGET_FREQ = 5 – minimum frequency in target corpus to be considered -FILE1 and FILE2 – change input file names if needed

Run the Script

bash python keyword_comparator.py

Output

Results are saved to keyword_comparison.xlsx with two sheets.

Requirements

• Python 3.x
• pandas
• nltk
• openpyxl

Install all dependencies with:

pip install -r requirements.txt