E-commerce Behavior Data Analysis

This project analyzes e-commerce behavior data using Apache Spark to extract meaningful insights about customer behavior, product performance, and sales patterns. The analysis is performed on two months of data (October and November) and includes both optimized and non-optimized implementations for performance comparison.

Project Overview

The project analyzes e-commerce data with the following key metrics:

• Product revenue analysis
• Conversion rates
• User visit patterns
• Age group analysis
• Gender-based analysis
• User retention rates

Key Features

• Dual implementation (optimized and non-optimized) for performance comparison
• Data exploration using Python/Jupyter notebooks
• Scala-based Spark processing
• AWS S3 integration for data storage
• Comprehensive metrics calculation
• Performance logging and timing

Dataset

Name: eCommerce Behavior Data from Multi-Category Store
Source: Kaggle
Size: ~15GB
Description: The dataset contains user interactions such as views, add/remove to/from cart and purchases on various products across multiple categories. The data includes timestamps, user IDs, session IDs, product IDs, and categories, making it suitable for detailed behavioral analysis.

Project Structure

├── src/
│   ├── main/
│   │   ├── python/
│   │       └── data_exploration.ipynb           # Data exploration notebook
│   │   ├── scala/
│   │   │   ├── app/
│   │   │   │   ├── EcommerceAnalysisApp.scala    # Main application
│   │   │   │   └── EcommerceAnalysisParser.scala # Data parsing logic
│   │   │   └── utils/
│   │   │       ├── Commons.scala                 # Common utilities
│   │   │       └── Config.scala                  # Configuration
│   └── outputs/                                 # Performance logs

Data Processing Jobs

1. Product Revenue Analysis

• Description: Analyze total revenue generated by products, identifying top and least performing items
• Output: Top and bottom products by revenue, including category and brand information

2. Product Conversion Rate Analysis

• Description: Calculate view-to-purchase conversion rates for all products
• Output: Products with highest and lowest conversion rates, including category and brand details

3. User Visit Analysis

• Description: Analyze user engagement through visit counts and patterns
• Output: Average visit counts per user and across all users

4. Age Group Analysis

• Description: Study conversion rates across different age segments (18-24, 25-34, 35-44, etc.)
• Output: Conversion rates by age group to identify high-performing demographics

5. Gender-Based Analysis

• Description: Analyze conversion rates and behavior patterns by gender
• Output: Gender-specific conversion rates and engagement metrics

6. User Retention Analysis

• Description: Calculate user retention rates based on repeat visits
• Output: Overall retention rate and returning user statistics

Setup and Installation

1. Prerequisites

   • Apache Spark
   • Scala
   • Python (for data exploration)
   • AWS account with S3 access
   • JDK 11

2. AWS Configuration

   • Create a file src/main/resources/aws_credentials.txt with your AWS credentials:

     [your-access-key]
     [your-secret-key]
     

   • Update src/main/scala/utils/Config.scala with your:
     • Project directory path
     • S3 bucket name

3. Data Preparation for Local Execution

   • Upload the October and November datasets to your configured S3 bucket though the python notebook dedicated cell:

       if not os.path.exists('../../../datasets/2019-Nov.csv') and os.path.exists('../../../datasets/2019-Oct.csv'):
           kaggle.api.authenticate()
           kaggle.api.dataset_download_files('mkechinov/ecommerce-behavior-data-from-multi-category-store', path='../../../datasets', unzip=True)
   
           df = pd.read_csv('../../../datasets/2019-Nov.csv')
           df = df.sample(frac=0.5)
           df.to_csv('../../datasets/2019-Nov.csv', index=False)
   
           df = pd.read_csv('../../../datasets/2019-Oct.csv')
           df = df.sample(frac=0.5)
           df.to_csv('../../datasets/2019-Oct.csv', index=False)

Remote Execution

Prerequisites

• AWS CLI configured
• EMR cluster created and running
• Security group configured for SSH access
• AWS credentials file set up in src/main/resources/aws_credentials.txt
• Create gradle app jar file with ./gradlew, the file will be located in build/libs/ECommerceAnalysis.jar

Run the Analysis

• Select "Spark Cluster" configuration
• Use argument "opt" for optimized version or "non-opt" for non-optimized version
• Click Run/Debug to execute

Performance Comparison

Based on local execution times:

• Optimized version: ~97 seconds
• Non-optimized version: ~129 seconds

The optimized implementation achieves better performance through:

• Combined processing of both months
• Efficient RDD caching
• Optimized transformations

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

The GNU GPL is a free, copyleft license that ensures the software remains free and open source. Users are free to:

• Use the software for any purpose
• Change the software to suit their needs
• Share the software with others
• Share the changes they make

For more information about the GNU GPL v3.0, visit https://www.gnu.org/licenses/gpl-3.0.html