A lightweight, modular Python utility designed to fetch raw text data from the Wikipedia API and apply various text processing, cleaning, and normalization techniques. Perfect for preparing NLP datasets, text mining, or generating clean corpora for machine learning models.
- Wikipedia API Integration: Seamlessly fetch page content, summaries, or categories using the official MediaWiki API.
- Robust Text Cleaning: Removes Markdown, HTML tags, Wikipedia syntax (like
{{citations}},[[links]]), and extraneous whitespace. - NLP Preprocessing: Built-in pipelines for tokenization, lemmatization, stopword removal, and lowercasing.
- Export Formats: Save processed text directly to
.txt,.json, or.csvfiles. - Rate-Limit Friendly: Follows Wikimedia's API etiquette with proper User-Agent headers and request throttling.
Clone the repository and install the required dependencies:
git clone [https://github.com/yourusername/wikipedia-text-processor.git](https://github.com/yourusername/wikipedia-text-processor.git)
cd wikipedia-text-processor
pip install -r requirements.txt