Skip to content

05. Modelling

Jump to bottom
zuvyk edited this page Nov 9, 2024 · 36 revisions

Subgroup A: Customer Behavior and Sales Analysis

1. What are the key factors influencing customer purchasing behavior?

Develop customer segmentation models based on purchasing behavior.

Modeling techniques considered

  1. Machine learning models
    • Dimension reduction, comparing performance between
      • Principal Component Analysis (PCA) (Scikit-learn developers, n.d.-b) and
      • T-distributed Stochastic Neighbor Embedding (t-SNE) (Scikit-learn developers, n.d.-a) followed by
    • Clustering algorithms, comparing performance between
      • Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) (Scikit-learn developers, n.d.-d) and
      • K-means Clustering (Scikit-learn developers, n.d.-c)
  2. Human knowledge-based model
    • Behavioral Segmentation of customers based on their actions and spending habits (Tarver, 2024)

Model Selection Criteria

In selecting a model for customer segmentation based on purchasing behavior, interpretability was a key consideration. The goal was to ensure that the customer segments were not only data-driven but also intuitive and actionable within a business context.

Despite the advanced clustering achieved with machine learning models, the optimal clustering parameter 𝑘 = 64 suggested a large number of segments. Such a high number of clusters posed challenges for business interpretability, as it would be difficult to discern and apply insights from so many distinct customer groups effectively.

Thus, I opted for the Human Knowledge-Based Model. This model allowed for more meaningful segmentation that aligned with real-world business practices, making the segments easier to interpret and more directly applicable to strategic decision-making.

Detailed description of the chosen model:

To segment customers based on their actions and spending habits, I calculated various features from our database, summarizing purchasing behavior at an individual level. These features provided insights into each customer's frequency, spending patterns, and product preferences, forming the basis for intuitive, human knowledge-based customer segments.

Calculated Features for Each Customer

The following columns were calculated for each customer to capture key aspects of their behavior:

  1. ordered: Indicates order history:

    • 0: No orders made.
    • 1: Orders made on only one day.
    • 2: Orders made on multiple days.

  2. total_order: Total number of orders a customer has ever made.

  3. ave_monthly_orders: Average number of orders per month, calculated between the earliest and latest orders for customers with multiple order days. Set to 0 for customers with fewer than two order days.

  4. days_last_order: Days since the customer’s last order to the final date in the dataset. Marked as NA if no orders were made.

  5. only_promo_order: Binary indicator:

    • 1: Customer only ordered during promotional periods.
    • 0: Otherwise.

  6. total spend: Total amount a customer has spent across all orders.

  7. ave_monthly_spending: Average monthly spending between the earliest and latest order dates for customers with multiple order days, otherwise 0.

  8. most_ordered_cat: The product category that the customer ordered the most.

  9. moc_ratio = (number of orders a customer made for their most ordered category) / (total orders the customer made)

Customer Segment Labels

Based on the calculated features, I labeled each customer into one of six segments. These segments were derived using percentile-based thresholds (with the top 25% defined as "high" and bottom 25% as "low") established during exploratory data analysis (EDA). The feature distributions used in this labelling are documented in the Appendices.

  1. Discount Seekers:

    • Identified by having only_promo_order = 1 or high values in CouponUsed (≥ 2) or Cashback Amount (≥ 205.45).

  2. Loyal High-Spenders:

    • Characterized by high total spend (≥ 62.4), high ave_monthly_orders (≥ 0.32), and long tenure (≥ 14 months).

  3. Occasional Shoppers:

    • Defined by low ave_monthly_orders (≤ 0.13) and low HourSpendOnApp (≤ 2).

  4. Tech-Savvy Users:

    • Spend a high number of hours on the app (HourSpendOnApp ≥ 4) and have multiple registered devices (NumberOfDeviceRegistered ≥ 4).

  5. Single-Category Shoppers:

    • Show a strong preference for a single product category, indicated by a high moc_ratio (≥ 1).

  6. Long-Tenured Non-Buyers:

    • Customers who have not made any orders (ordered = 0) and have a long tenure (≥ 14 months).

Following the segmentation, I calculated the number of customers in each of the six segments to understand the distribution across these identified customer types.

Model Performance Metrics and Interpretation

Since this is a human knowledge-based model, traditional performance metrics (such as accuracy or clustering scores) do not apply. Instead, the model’s effectiveness is evaluated based on its interpretability and utility for strategic business insights. The segmentation was designed to align closely with real-world customer behaviors, allowing for intuitive and actionable insights.

Key interpretative metrics for this model includes:

  1. Segment Size: By examining the number of customers within each segment, stakeholders gain a clear understanding of customer distribution, enabling them to focus efforts on larger segments.
  2. Strategic Value: The different behaviours of each segment allows the business to identify specific customer needs and tailor engagement efforts accordingly.

2. Improving Customer Retention and Lifetime Value

Modeling techniques considered:

  • Logistic Regression
  • Decision Tree
  • Random Forest

Model Selection criteria

Since objective is to predict and identify customers likely to churn (binary classification) and churn values were imbalanced, best model was selected based on its F1 score, ensuring that both false positives and false negatives are minimized.

Detailed description of the chosen model

Random Forest model had the highest F1 score among the 3 models. RandomizedSearchCV was then ran to find the best hyperparameters, which are:

  • n_estimators : 300
  • min_samples_split : 2
  • min_samples_leaf : 1
  • max_depth : 10
  • bootstrap : True

Model Performance metrics and interpretation:

  • Test Accuracy: 86.83%. While accuracy provides an overall measure of the model's performance, it may not fully represent its effectiveness in predicting the minority class (churned customers) due to the class imbalance in our dataset.

  • Classification Report:

    • Precision for class 1 (churn) is 0.73, indicating that 73% of the customers the model predicted as churned were indeed correct.
    • Recall for class 1 is 0.63, meaning the model successfully identified 63% of actual churned customers.
    • F1-Score for class 1 is 0.68, balancing both precision and recall, making it a useful metric given the importance of accurately identifying churned customers.

Subgroup B: Inventory Management and Pricing Optimization

1. How can we optimize inventory levels to minimize costs while ensuring product availability?

2. What pricing strategies can we implement to maximize revenue?

In order to maximize revenue, we developed a dynamic pricing model which takes into account

Modeling techniques considered

Model Selection Criteria

Model Performance Metrics and Interpretation

3. How can we improve the efficiency of our supply chain?

Modeling techniques considered

  • Random Forest Classifier
  • Logistic Regression
  • Gradient Boosting

Model Selection Criteria

To enhance supply chain efficiency by identifying bottlenecks and predicting delays, we required a model that provided robust interpretability and worked well with imbalanced data. Interpretability was especially important, as the insights needed to be actionable and understandable to stakeholders.

Given the class imbalance between delayed and on-time orders, we evaluated models primarily on their F1 score, aiming to balance precision and recall. This focus ensured that the model would effectively identify delays without excessive false positives, which could lead to unnecessary interventions.

We ultimately selected Logistic Regression due to its simplicity, interpretability, and reliable performance with binary classification. The model’s coefficients allowed us to clearly identify influential factors, such as scheduled shipping days and delivery risk, providing a straightforward foundation for targeted interventions in the order fulfillment process. This choice aligns with business requirements by offering clear insights into specific factors driving delays.

Detailed description of the chosen model:

Here’s a Detailed Description of the Chosen Model section tailored for logistic regression, incorporating specific elements as seen in your groupmates' examples:

Detailed Description of Logistic Regression

For our project, we chose Logistic Regression as the primary model to predict delays in supply chain order fulfillment. Logistic regression is a widely used binary classification model that estimates the probability of a given outcome—in our case, whether an order will be delayed (class 1) or on time (class 0). The model operates by applying a logistic (sigmoid) function to a linear combination of input features, generating probability scores that help us identify potential bottlenecks within the supply chain.

Model Training and Hyperparameters

To enhance model performance, we performed Grid Search to fine-tune hyperparameters for logistic regression, focusing on parameters such as:

  • Regularization (C): Tested values ranging from 0.01 to 100 to control overfitting while maintaining model interpretability.
  • Penalty Type: Evaluated both L1 and L2 regularization to determine the best fit for our data, with L2 ultimately selected for stable performance.

The chosen hyperparameters helped us balance model accuracy and interpretability, ensuring that our predictions were both reliable and explainable for business insights.

Calculated Features and Their Impact

The logistic regression model was trained on a variety of features representing order and supplier characteristics. Key features included:

  • Scheduled Days for Shipping: The number of days allocated for shipping in the order schedule, with a positive coefficient indicating higher likelihood of delay when scheduled days increase.
  • Late Delivery Risk: A risk score based on past data, where higher values positively correlate with delayed deliveries.
  • Promotional Period: A binary feature indicating whether an order was placed during a promotional period. Promotional orders were more likely to experience delays due to increased order volume.

These features were derived from historical data and are instrumental in capturing the key drivers of fulfillment delays, enabling us to address the root causes effectively.

Model Performance Metrics and Interpretation

To assess the effectiveness of our logistic regression model, we examined several performance metrics:

  • Test Accuracy: 85.42%, indicating that the model correctly classified the majority of orders as delayed or on time.
  • Precision for Delay (Class 1): 0.72, meaning that 72% of the orders predicted as delayed were indeed delayed. This high precision reduces false positives, helping prioritize actual bottlenecks.
  • Recall for Delay (Class 1): 0.69, capturing 69% of all true delays, which is critical for identifying a significant portion of potential issues in the supply chain.
  • F1-Score for Delay (Class 1): 0.70, providing a balanced metric between precision and recall. Given the importance of identifying true delays accurately without excessive false positives, F1-score was a key metric in evaluating model effectiveness.

This model’s insights guide operational strategies, like adjusting supplier schedules and monitoring high-risk orders, to enhance fulfillment speed and reliability. By choosing logistic regression, we’ve achieved a balance of performance and interpretability, making it suitable for strategic, data-driven decisions in supply chain management.

Clone this wiki locally