DataStep

A Python tool for analyzing insurance data across two time periods. It helps insurance companies validate their data, find outliers, track records coming on and going off the book, and ensure data consistency.

Features

• Load and compare data from two time periods
• Find outliers in numeric and categorical data
• Track new and lapsed policies
• Validate business rules (dates, premiums, etc.)
• Optional LLM-based data validation using Ollama
• Generate reports in CSV, JSON, and HTML formats

Requirements

pip install polars requests jinja2

You'll need Ollama installed and running if you want to use the LLM validation features.

Setup

1. Clone the repository
2. Create these folders in your project:
   • data/ (for your CSV files)
   • reports/ (for output)
   • templates/ (for HTML templates)
3. Copy your data files to the data/ folder
4. Update config.json with your settings

Files

• main.py - Main script to run the analysis
• data_loader.py - Handles data loading and initial validation
• outlier_detector.py - Finds outliers in the data
• data_comparison.py - Compares datasets and tracks changes
• business_validator.py - Validates business rules
• llm_validator.py - Optional LLM-based validation
• report_generator.py - Generates reports
• config.json - Configuration settings

Usage

1. Update config.json with your settings

2. Run:

   python main.py

Output

• CSV files with detailed analysis
• JSON summary of findings
• HTML report with key metrics
• Validation results
• Lists of new and lapsed policies

Notes

• Designed for large datasets (1M+ records)
• Uses Polars for efficient data processing
• Configurable for different insurance companies

Future developments

• Implementing line-by-line data comparison and generating a summary that highlights the percentage of records changed for each field. This can be enhanced further: for numerical fields, categorize changes into increases and decreases; for categorical fields, show the distribution in the previous dataset versus the current one.
• Enhancing LLM-based validation capabilities.

Contributing

Feel free to open issues or submit pull requests.