Big Data Engineering — Project 2
data-rush/
├── docker-compose.yml # MongoDB container
├── .env # Environment variables
├── spark/
│ ├── jobs/
│ │ ├── peek.py # Explore dataset schema
│ │ ├── phase1.py # MapReduce: market basket + user affinity
│ │ ├── phase2.py # Load Phase 1 results into MongoDB
│ │ └── phase3.py # Cart abandonment + recommendation labels
│ └── data/
│ ├── ecommerce_logs.csv # ← place dataset here
│ └── output/ # generated by phase1.py
│ ├── market_basket.csv
│ ├── user_affinity.csv
│ └── cart_abandonment.csv
├── query_demo.py # Interactive query demo (Phase 2/3 deliverable)
└── schema_document.pdf # NoSQL schema design document
| Tool | Version | Download |
|---|---|---|
| Python | 3.11 or 3.12 | https://www.python.org/downloads/ |
| Java JDK | 17 only | https://adoptium.net/temurin/releases/?version=17 |
| Docker Desktop | Latest | https://www.docker.com/products/docker-desktop/ |
| Git | Any | https://git-scm.com/ |
⚠️ Java 17 is required. Java 21+ and Java 23 cause PySpark to fail.
Download both files and place in D:\winutils\bin\:
- https://github.com/cdarlint/winutils/raw/master/hadoop-3.3.5/bin/winutils.exe
- https://github.com/cdarlint/winutils/raw/master/hadoop-3.3.5/bin/hadoop.dll
Then set the system environment variable:
# Run as Administrator
[System.Environment]::SetEnvironmentVariable("HADOOP_HOME", "D:\winutils", "Machine")Also disable the Microsoft Store Python alias: Settings → Apps → Advanced app settings → App execution aliases → turn OFF python.exe and python3.exe
git clone <repo-url>
cd data-rushCopy ecommerce_logs.csv into:
spark/data/ecommerce_logs.csv
python -m venv venv
# Windows
venv\Scripts\activate
# Mac/Linux
source venv/bin/activatepip install pyspark pymongo pandasdocker compose up -d
docker compose ps # confirm mongo is runningOpen each of phase1.py, phase3.py and update these lines at the top to match your actual path:
os.environ["PYSPARK_PYTHON"] = r"C:\path\to\your\data-rush\venv\Scripts\python.exe"
os.environ["PYSPARK_DRIVER_PYTHON"] = r"C:\path\to\your\data-rush\venv\Scripts\python.exe"Run these scripts in order:
# Phase 1: MapReduce — market basket analysis + user affinity
python spark/jobs/phase1.py
# Output: spark/data/output/market_basket.csv
# spark/data/output/user_affinity.csv
# Phase 2: Load results into MongoDB
python spark/jobs/phase2.py
# Output: 25,000 user profiles + 65,000 item pairs in MongoDB
# Phase 3: Cart abandonment + discount labeling
python spark/jobs/phase3.py
# Output: spark/data/output/cart_abandonment.csv
# Query demo (interactive)
python query_demo.py| Script | Output | Description |
|---|---|---|
| phase1.py | market_basket.csv | Item pairs bought together with frequency |
| phase1.py | user_affinity.csv | Per-user category scores (view=1, cart=3, purchase=5) |
| phase2.py | MongoDB collections | user_profiles + market_basket with indexes |
| phase3.py | cart_abandonment.csv | High_Discount or Standard_Reminder per abandoned cart |
| query_demo.py | Terminal output | Live queries against MongoDB |
timestamp, session_id, user_id, event_type, product_id,
price, referrer, user_metadata, product_metadata
event_type: view, cart, purchaseproduct_metadata: JSON string containing category, brand, stock
Python worker failed to connect → Python alias conflict. Disable Microsoft Store Python alias (see Prerequisites).
HADOOP_HOME unset → winutils not configured. Follow the Windows winutils setup above.
mongo:27017 connection refused → Use localhost:27017 for local scripts (not mongo:27017 — that's only for Docker containers).
Path does not exist: ecommerce_logs.csv → Make sure the CSV is in spark/data/ and the filename in phase1.py matches exactly.
Docker not starting → Open Docker Desktop first, wait for the whale icon to stop animating, then run docker compose up -d.