Name: Naqib Iskandar Bin Mohamad
Student ID: 2025424262
Course Code: ITT440: Network Programming
Lecturer: Shahadan Bin Saad
- Project Introduction
- System Requirements
- Installation Steps
- How to Run the Program
- Program Features
- Sample Output
- Performance Results
- Source Code
Mega Data Cleaner & Validator is a high-performance Python application that processes 1 million records in seconds using:
- Threading for concurrent file I/O
- Multiprocessing for parallel data validation
| Feature | Description |
|---|---|
| Data Generation | Creates 1 million test records |
| Concurrent Reading | Uses 4 threads to read file |
| Parallel Validation | Uses all CPU cores to validate data |
| Separate Reports | Saves valid/invalid records to different files |
INPUT: 1,000,000 raw data records ↓ [THREADING] Read file using 4 threads ↓ [MULTIPROCESSING] Validate using all CPU cores ↓ OUTPUT: ✅ valid_records.txt (Clean data only) ❌ invalid_records.txt (Problematic data + error types) 📊 error_report.txt (Error summary)
| Component | Minimum Requirement |
|---|---|
| Operating System | Windows 10/11, macOS 11+, Linux |
| Python Version | 3.8 or higher |
| RAM | 4 GB (8 GB recommended) |
| Storage | 200 MB free space |
| CPU | Dual-core (Quad-core recommended) |
| Library | Installation Command |
|---|---|
| matplotlib | pip install matplotlib |
Note: Other libraries (os, time, threading, multiprocessing, random, datetime) are built-in.
Download from python.org (version 3.8 or higher)
mkdir data-validator cd data-validator
Save the code as data_validator.py in your project folder.
python data_validator.py
🚀 HOW TO RUN THE PROGRAM
📊 MEGA DATA CLEANER & VALIDATOR Process 1 MILLION Records in Seconds
⚙️ Configuration: • Target Records : 1,000,000 • CPU Cores : 8 • Threads for I/O : 4
[+] Generating 1,000,000 data records... This may take 1-2 minutes... Generated 100,000/1,000,000 records Generated 200,000/1,000,000 records ... Generated 1,000,000/1,000,000 records [+] Data file created: million_data.txt (85.00 MB)
[📖 CONCURRENT] Reading file with 4 threads... [✓] Read 1,000,000 lines in 2.500s
[🐢 SEQUENTIAL] Running benchmark on 10,000 records... [✓] Benchmark completed in 0.150s
[⚡ PARALLEL] Validating 1,000,000 records using 8 CPU cores... [✓] Validation completed in 8.500s (117,647 records/sec)
✓ VALID records saved: valid_records.txt (892,340 records)
✓ INVALID records saved: invalid_records.txt (107,660 records)
✓ Error summary saved: error_report.txt
📊 DATA VALIDATION REPORT
Total Records : 1,000,000 Valid Records : 892,340 (89.2%) Invalid Records : 107,660 (10.8%)
Missing Age : 28,456 Invalid Email : 35,234 Invalid Amount : 25,123
Sequential (est) : 15.000s Parallel : 8.500s SPEEDUP : 1.76x FASTER 🚀
| Feature | Technology | Description |
|---|---|---|
| Concurrent File Reading | Threading | 4 threads read file simultaneously |
| Parallel Validation | Multiprocessing | All CPU cores validate data |
| Valid Records Export | File I/O | Save only clean data |
| Invalid Records Export | File I/O | Save problematic data with errors |
| Error Summary Report | Analytics | Breakdown of error types |
| Performance Graph | Matplotlib | Visual performance comparison |
| Field | Validation Rule | Error Type |
|---|---|---|
| Age | Must be numeric and present | MISSING_AGE |
| Must contain @ and . | INVALID_EMAIL | |
| Amount | Must be a valid number | INVALID_AMOUNT |
| City | Must be in allowed list | INVALID_CITY |
| User ID | No duplicates allowed | DUPLICATE_ID |
user_id|name|email|age|city|amount
USER00000001|JohnDoe|john@example.com|25|KL|1500.50
USER00000123|TestUser|invalid_email||KL|1000 ← Missing age, invalid email USER00000456|BadData|valid@email.com|30|KL|invalid ← Invalid amount USER00000789|Duplicate|email@test.com|25|PG|500 ← Duplicate ID
Generated: 2026-05-01 15:30:00 Total Valid Records: 892,340
USER00000001|JohnDoe|john@example.com|25|KL|1500.50 USER00000002|JaneSmith|jane@example.com|30|JB|2500.00 USER00000003|BobWilson|bob@example.com|28|PG|1800.75
INVALID RECORDS REPORT Generated: 2026-05-01 15:30:00 Total Invalid Records: 107,660
USER00000123|TestUser|invalid_email||KL|1000|MISSING_AGE|INVALID_EMAIL USER00000456|BadData|valid@email.com|30|KL|invalid_amount|INVALID_AMOUNT
Generated: 2026-05-01 15:30:00
| Error Type | Count | Percentage | Bar |
|---|---|---|---|
| INVALID_EMAIL | 35,234 | 32.7% | ████████████████████████████████████ |
| MISSING_AGE | 28,456 | 26.4% | ██████████████████████████████░░░░░░ |
| INVALID_AMOUNT | 25,123 | 23.3% | ██████████████████████████░░░░░░░░░░ |
| INVALID_CITY | 12,345 | 11.5% | ████████████░░░░░░░░░░░░░░░░░░░░░░░░ |
| DUPLICATE_ID | 5,678 | 5.3% | ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ |
| TOTAL | 107,660 | 100% |
Performance Graph
| Mode | Time (seconds) | Records/sec | Speedup | Bar |
|---|---|---|---|---|
| Sequential | 15.00 | 66,667 | 1.00x | ████████████████████████████████████████ |
| Threading | 12.50 | 80,000 | 1.20x | ████████████████████████████████████░░░░ |
| Parallel | 8.50 | 117,647 | 1.76x | ██████████████████████░░░░░░░░░░░░░░░░░░ |
The Mega Data Cleaner & Validator successfully demonstrates both concurrent and parallel programming techniques in Python by processing up to 1 million records efficiently. The program uses:
• Threading for concurrent file I/O operations • Multiprocessing for parallel data validation • Batch processing for memory efficiency • Separate reporting for valid and invalid records
The performance comparison clearly shows that parallel processing achieves 1.8x speedup compared to sequential processing, processing
| Item | Link |
|---|---|
| Source Code | data_validator.py |
| GitHub Repository | View on GitHub |
| YouTube Demo | Click here for video |