A/B Testing & Uplift Modeling (Criteo Uplift Dataset)

This project shows an end-to-end experimentation workflow: validate the experiment, measure average impact, find who benefits most, and make a business decision. It’s written to be understandable to entry‑level data scientists while still using senior‑grade rigor.

Overview

• Dataset: randomized ad experiment with treatment/control and exposure.
• Primary goal: measure conversion lift and decide whether to ship.
• Secondary goal: identify high‑uplift users for personalized targeting.

Key Results (from the executed notebooks)

• Conversion lift: +65% relative (+0.13pp absolute), p < 0.001
• Visit lift: +21% relative
• Post‑hoc power: 99.8%
• Value concentration: top 40% of users drive ~70% of uplift
• Recommendation: ship with a phased rollout
• 3‑month impact (conservative): ~3,900 conversions, ~$292.5K revenue, ~$6K cost, ~$286.5K net profit

What This Project Covers

• Proper A/B test analysis with confidence intervals and significance testing.
• Statistical vs practical significance.
• Power analysis (pre and post).
• Segmentation and heterogeneity.
• Uplift modeling and evaluation (AUUC, Qini).
• Clear decision: ship, target, or stop.

Project Structure

AB_test/
├── README.md
├── notebooks/
│   ├── 01_eda_and_experiment_design.ipynb
│   ├── 02_ab_test_analysis.ipynb
│   ├── 03_uplift_modeling.ipynb
│   └── 04_targeting_policy_and_business_decision.ipynb
├── reports/
│   ├── executive_summary_findings.md
│   ├── findings_and_recommendations.md
│   └── learnings_readme.md
├── src/
│   ├── data_prep.py
│   ├── metrics.py
│   └── models.py
└── requirements.txt

How to Run

1. Create and activate a virtual environment:

cd /Users/avikshithreddy/Desktop/AB_test
python -m venv .venv
source .venv/bin/activate

2. Install dependencies:

pip install -r requirements.txt

3. Open VS Code and run the notebooks in order:
4. notebooks/01_eda_and_experiment_design.ipynb
5. notebooks/02_ab_test_analysis.ipynb
6. notebooks/03_uplift_modeling.ipynb
7. notebooks/04_targeting_policy_and_business_decision.ipynb

Note: The notebooks default to a small sample (sample_frac=0.02) for speed. Increase for full‑scale runs.

Decision Logic (Simple Version)

• Go if the 95% CI for lift is positive and ROI is positive.
• Targeted go if the top deciles show strong uplift even when the global lift is modest.
• No‑go if confidence intervals cross zero or ROI is negative across cutoffs.

Assumptions Used for ROI

• CPM: $2.00
• Conversion value (AOV): $75.00
• Treatment ratio: ~85%

What to Read

• reports/executive_summary_findings.md for a 1‑page summary.
• reports/findings_and_recommendations.md for the full technical write‑up.
• reports/learnings_readme.md for a simple one‑page summary of learnings and impact.

Tools and Libraries

• pandas, numpy, scikit‑learn
• statsmodels/scipy for testing and power
• causalml for X‑Learner and DR‑Learner
• scikit‑uplift for uplift evaluation utilities