Data Validation Framework

A Python-based framework for comparing source and target CSV files to identify data discrepancies and generate comprehensive validation reports.

Overview

This framework provides automated data validation by comparing CSV files from source and target directories. It identifies differences in data, structure, and quality, generating actionable reports for data reconciliation.

Features

• Schema Validation: Compare column names, data types, and structure
• Row Count Validation: Detect missing or extra rows between source and target
• Data Comparison: Identify row-level and column-level discrepancies
• Duplicate Detection: Flag duplicate records in datasets
• Null/Empty Analysis: Identify missing or null values
• Summary Reports: Generate human-readable validation summary with detailed discrepancy breakdown
• CSV Export: Export validation results for further analysis

Project Structure

data_validation_framework/
├── README.md
├── requirements.txt
├── config/
│   └── validation_config.yaml
├── src/
│   ├── __init__.py
│   ├── validator.py
│   ├── report_generator.py
│   └── utils.py
├── tests/
│   ├── __init__.py
│   ├── test_validator.py
│   └── test_report_generator.py
├── data/
│   ├── source/
│   │   └── source.csv
│   └── target/
│       └── target.csv
└── examples/
    └── example_usage.py

Installation

# Clone the repository
git clone https://github.com/bbhuwanbhatt/data_validation_framework.git
cd data_validation_framework

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Quick Start

from src.validator import DataValidator

# Initialize validator with source and target directories
validator = DataValidator(source_dir='data/source', target_dir='data/target')

# Run validation
results = validator.validate()

# Generate report
validator.generate_report(output_path='validation_report.txt')

# Print summary
print(validator.summary)

Configuration

Edit config/validation_config.yaml to customize validation behavior:

comparison:
  case_sensitive: false
  ignore_whitespace: true
  null_as_empty: true

reporting:
  max_discrepancies_shown: 100
  include_duplicates: true
  include_nulls: true

Validation Reports

The framework generates:

• Validation Summary: High-level metrics (total rows, matches, mismatches, etc.)
• Schema Differences: Column additions, removals, type mismatches
• Row-Level Discrepancies: Specific rows with differences
• Data Quality Issues: Duplicates, null values, inconsistencies

Command Line Usage

python examples/example_usage.py

Testing

# Run all tests
pytest tests/

# Run with coverage
pytest --cov=src tests/

API Reference

DataValidator

Main class for data validation operations.

Methods:

• validate(): Execute full validation pipeline
• generate_report(output_path): Generate text report
• get_summary(): Return validation summary dict
• export_results(csv_path): Export results to CSV

ReportGenerator

Handles report formatting and output.

Methods:

• generate_text_report(): Create formatted text report
• generate_html_report(): Create HTML report (future)
• format_discrepancies(): Format discrepancy details

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/your-feature)
3. Commit changes (git commit -am 'Add feature')
4. Push to branch (git push origin feature/your-feature)
5. Create a Pull Request

License

MIT License - see LICENSE file for details

Support

For issues or questions, please open an issue on GitHub.