CSV Processor (Python)

A small Python utility for loading CSV data from a configured directory, standardizing column names, removing duplicates, checking for missing values, and saving a cleaned CSV output.

Features

• Load CSV files from a directory configured via environment variables.
• Standardize column names to snake_case.
• Remove duplicate rows.
• Report missing values and halt processing if any are found.
• Save cleaned data to data/processed with timestamped filenames.

Project Structure

.
├── data/
│   └── raw/
├── src/
│   ├── app.py
│   └── main.py
├── .env
├── requirements.txt
└── README.md

Requirements

• Python 3.9+
• Dependencies in requirements.txt

Setup

1. Create and activate a virtual environment (optional but recommended).
2. Install dependencies:

pip install -r requirements.txt

3. Configure the data directory in .env:

DIRECTORY=/absolute/path/to/your/raw/data

Place your CSV files in that directory (or update the path accordingly).

Usage

Run the main entrypoint:

python src/main.py

The program will:

1. Load CSV files from the configured DIRECTORY.
2. Standardize column names.
3. Remove duplicate rows.
4. Check for missing values (halts if any are found).
5. Save cleaned data to data/processed with a timestamped filename.

Output

Cleaned files are saved to:

data/processed/cleaned_data_<timestamp>.csv

Notes

• Only .csv files are currently loaded in the implementation, even though .json and .xlsx are listed as acceptable extensions.
• If no files are found or no valid data is loaded, the script exits with a log message.

License

See LICENSE.