Task 2 — Pre-Project Data Profiling & Audit

PlaceMux · Altrodav Technologies Pvt. Ltd. · Phase 1 Industry Immersion · Data Analyst track

This is a complete, runnable Python project that profiles and audits a realistic, deliberately-messy e-commerce orders dataset (8,240 rows, 18 columns), exactly as required by the task brief and study guide.

Everything here is generated by actually running the code — the charts, tables, and issue list in outputs/ are not hand-written; they are the real output of src/run_all.py on data/customer_orders_raw.csv.

---

Quick start

pip install -r requirements.txt

# (re)generate the messy dataset from scratch (optional — a copy is already included)
python data/generate_data.py

# run the full profiling + audit pipeline
python src/run_all.py

# run the edge-case / failure-handling test suite
python -m pytest tests/ -v

Then open outputs/dashboard.html directly in a browser — it's a single self-contained file (charts embedded as base64), so it needs no server and can be opened/demoed live, including offline.

---

Project structure

data/
  generate_data.py         # builds the realistic messy dataset (documented issue injection)
  customer_orders_raw.csv  # the dataset itself (8,240 rows x 18 cols)
  issue_manifest.json      # ground-truth record of every issue deliberately injected

src/
  utils.py                 # logging + safe_step() graceful-failure wrapper
  data_profiler.py         # DataProfiler: types, missingness, distributions, duplicates, categorical consistency
  quality_audit.py         # turns profiling results into a prioritised, severity-scored issue list
  report_builder.py        # writes profiling_report.md and audit_note.md
  dashboard_builder.py     # writes the single-file, live-demoable dashboard.html
  run_all.py                # ONE command that runs everything end-to-end

tests/
  test_edge_cases.py       # 8 tests proving graceful handling of missing files, empty data,
                            # all-null columns, bad dtypes, missing PK column, tiny datasets, etc.

outputs/                   # generated by running src/run_all.py (already populated)
  profiling_report.md
  data_quality_issues.csv
  audit_note.md
  dashboard.html
  charts/*.png
  pipeline.log

---

How this maps to the marking rubric (100 marks)

#	Scoring parameter	Marks	Where to look
1	Profiling report covering types, missingness, distributions and duplicates	25	outputs/profiling_report.md (sections 1-4) + outputs/charts/*.png + dashboard.html sections 1-4
2	Prioritised list of data-quality issues with proposed fixes	25	outputs/data_quality_issues.csv (17 issues, severity-ranked) + outputs/audit_note.md (narrative reasoning per issue)
3	Real-data quality & correctness (realistic scale, not toy/happy-path)	20	data/generate_data.py + data/issue_manifest.json — 8,240 rows with 10 documented, realistic issue types (missingness, dup PKs, inconsistent labels, outliers, mixed types, mixed date formats)
4	Live verification & evidence (demonstrated live; real output, not claims)	15	outputs/dashboard.html — open it, every number is computed from the actual dataset on the actual run; outputs/pipeline.log timestamps the real execution
5	Dependency, failure & edge-case handling (errors handled; hand-offs honoured)	15	src/utils.py (safe_step) used throughout data_profiler.py; tests/test_edge_cases.py — 8 passing tests for missing file, empty CSV, all-null column, non-numeric-in-numeric column, missing PK column, 1-row dataset, full real-dataset integration run

Total: 100

---

What the dataset deliberately contains (and why)

data/generate_data.py builds the CSV from scratch with a fixed random seed (reproducible) and documents every injected issue in data/issue_manifest.json:

• Missingness at different rates across 6 columns, with different causes (structural / MNAR vs. incidental / MCAR) so the audit has to reason about why, not just count nulls.
• 121 exact duplicate rows (simulated double-submitted forms).
• Duplicate primary keys with conflicting data — order_id repeats with a different total_amount, the classic "looks like a key but isn't" trap.
• Inconsistent categorical labels — Bangalore / Banglore / BLR / Bengaluru, Bombay / Mumbai, Credit Card / credit card / CC, etc. — exactly the example given in the task brief.
• Outliers & impossible values — negative quantity, ages of -5, 150, 999, unit prices in the ₹1-1.5 lakh range (decimal/currency entry errors), and total_amount spikes that don't reconcile with price × quantity.
• Mixed types — quantity sometimes stored as text ("two", "N/A").
• Inconsistent date formats — order_date mixes ISO and DD/MM/YYYY in the same column.

Running the pipeline recovers essentially all of these programmatically — see outputs/data_quality_issues.csv for the detected list.

---

Design notes

• Why a single-file HTML dashboard and not a Streamlit/Flask app? So it's demoable anywhere, instantly, with zero setup — a marker (or anyone) can just double-click dashboard.html. No server, no port, no "it works on my machine."
• Why safe_step? In a real pipeline, one bad column shouldn't take down the whole audit. Every analysis stage is independently wrapped; failures are logged and surfaced in the report itself (see the "Pipeline Reliability" section of the dashboard and report) rather than silently hidden or fatally crashing.
• Severity model for the prioritised issue list is spelled out in audit_note.md (HIGH/MEDIUM/LOW) so the prioritisation is defensible, not arbitrary.