-
Notifications
You must be signed in to change notification settings - Fork 0
11. Lessons Learned
One of the primary challenges in developing our customer segmentation model was the limited coverage of segmentation techniques in our lectures. While our course included a general overview of machine learning, the emphasis was largely on machine-based models rather than specifically on segmentation. The only machine learning model that seemed to be suitable for segmentation was unsupervised learning that was in the summary slide, with minimal guidance on how to apply these models for customer segmentation. This gap meant that I needed to independently research and study various unsupervised learning techniques to understand and implement an effective segmentation approach.
After testing several times with different unsupervised learning methods, I identified an optimal number of clusters as k=64, based on standard unsupervised clustering evaluation metrics. However, this result presented a second major challenge: from a business perspective, such a high number of clusters was impractical and extremely challenging to interpret. This outcome highlighted the need for alternative segmentation approaches that align better with business goals, which had not been covered in our lectures.
To overcome these challenges, I sought advice from our teaching assistant (TA), Hera, in a consultation session. Hera introduced me to a human knowledge-based approach for customer segmentation, where clusters could be designed around human-defined behavioral attributes. This approach provided a more practical, interpretable solution for our project and was highly effective in creating meaningful customer segments based on business insights.
One of the main challenges encountered was the insufficient resources and data available for the current retention strategy. To address this, we had to research common analytical methods for retention strategies and leverage synthetic data to fill in the gaps. However, it was crucial to ensure that the synthetic data accurately modelled real-life customer behavior, so retention rates from the current dataset were used as a benchmark.
Another difficulty was ensuring that our analysis was robust and well-supported by evidence. This required constant refining and experimentation with different approaches. Through iterative testing, we were able to develop a comprehensive analysis that provided actionable insights for our retention strategy.
One of the primary challenges we encountered during the project was the inadequacy of the dataset we found to thoroughly evaluate the impacts on both marketing channels and campaigns. This limitation forced us to utilize two separate datasets - one for campaigns and another for messages, rather than relying on a single comprehensive dataset.
The primary challenge was that the datasets did not provide a holistic view of the marketing efforts. The gaps in data left us unable to analyze key performance indicators effectively, hindering our assessment of impacts of marketing channels and campaign effectiveness. Furthermore, the messages dataset was considerably large, which caused our code to execute slowly and made it difficult to visualize the data efficiently. This performance issue significantly impacted our ability to conduct timely analysis and draw actionable insights.
To address these challenges, we implemented a solution that involved sampling a subset of campaigns from both the campaigns and messages tables. This approach allowed us to reduce the size of both datasets, facilitating a more manageable analysis without compromising the integrity of our findings.
However, after performing random sampling, we realized that the resulting tables were still insufficient for our analysis. We lacked essential product details, such as products, product costs and prices, which were critical for understanding the overall effectiveness of our marketing strategies.
To overcome this shortfall, we resorted to generating synthetic data to complement our analysis. We conducted thorough research in generating synthetic data, ensuring that the simulated product details accurately reflected real market patterns. We made sure to reference credible sources during this process to reduce potential biases and enhance the validity of the synthetic data.
By navigating these challenges with strategic solutions, we were able to refine our analysis, ultimately leading to more insightful conclusions regarding the impacts of marketing channels and campaigns. The use of synthetic data not only addressed the missing information but also helped create a more robust framework for evaluating marketing effectiveness.
-
Difficulty in Capturing Seasonality with ARIMA: ARIMA models often struggle to accurately capture seasonal patterns in data, as they are generally more suited to non-seasonal, stationary time series. Hence, to enhance seasonality detection, we used a seasonal variant of ARIMA, which incorporates additional seasonal components. SARIMA's seasonal differencing helped to capture periodic trends better.
-
Inability to Integrate Demand Forecasting into Linear Programming: translating demand forecasts into a linear programming model was difficult due to the variability and non-linearity of real-world demand data. Thus, to simplify the integration, we calculated monthly demand averages, which provided a stable, representative value for each period. We then incorporated common industry values such as holding and backorder costs from online sources.
-
Difficulty in visualising distribution of PED values: Due to the sheer amount of products (>20,000) in our dataset, there was quite a big variation in the PED values. The initial distribution was so skewed that it was difficult to draw any sound analysis. We overcame this by visualising the log transformation instead in order to see the general distribution of PED values without being affected by extreme values. This taught us the importance of data augmentation when attempting to draw insights.
-
A few items with high prices were affecting the pricing model's ability to generalise: This caused the root mean squared error (RMSE) of the initial pricing model to be very high, more than 200% of the average price of products. This basically rendered the model useless, and we would be better off without it. To address this, we simply removed products in the top and bottom 5 percentile of prices. This allowed the model to generalise better, reducing the RMSE to 50% of the average price of products. It was a trade-off between achieving better model performance for the majority of products versus having poor model performance across all products.
-
Using Resampling for Time-Based Insights: To analyze trends in shipping duration, we applied resampling to calculate monthly averages. This helped us understand seasonal fluctuations and detect patterns in delivery times. This taught us the value of time-based resampling in gaining temporal insights and monitoring performance over consistent intervals.
-
Evaluating Model Performance with Multiple Metrics: We used a range of metrics, including RMSE and F1-score, to evaluate model performance comprehensively. Initially, accuracy alone did not provide a full picture, especially for predicting delays. By expanding our evaluation with additional metrics, we gained a clearer understanding of the model's strengths and limitations in handling different delivery scenarios.
Customer review sentiment analysis has the biggest issue of finding a model appropriate to work with the reviews. In our project, the reviews were messy and incoherent. Our decision was to stick with simpler models, as they may be better than advanced ones, which tend to overfit data. As the data input is noisier/messier, we thought that simpler models are just as capable as their advanced counterparts for performing the required language tasks. Furthermore, a simpler, slightly less accurate model is a pragmatic choice for this as it is faster and more robust to handle messy data.
As mentioned in our recommendations of our features, we are open to considering replacing our current model with a more advanced one. However, deeper research has to be done before carrying out this phase.
Another minor challenge is to decide on extracting issues from all reviews or from just the negative and neutral reviews. With our model (NER with SpaCy), we extracted a very long list of negative entities from all reviews, with many entities not considered to be negative or relevant to our expected insights. However, extracting negative entities from negative and neutral reviews only provides us with a shorter list of several negative entities obtained. Hence, we went with the latter approach. While this is not an accurate way to extract common issues as we only obtain from a subset of all reviews, it was likely the best we could come up with, due to the vagueness and disorderly nature of the reviews. In the future, steps should be taken to improve customer review input to provide us with less muddled reviews to work with.
This project allowed us to understand how to transform complex data analysis — such as graphs, statistical findings, and model outputs — into meaningful insights for non-technical stakeholders. Additionally, we gained a deeper understanding of the end-to-end pipeline of a data science project, from data preprocessing and analysis to developing models and presenting insights in a clear, digestible way. The experience also highlighted the importance of continuously refining the approach based on feedback and data insights to ensure relevance and accuracy in the findings.
In this project, sourcing datasets that could effectively address our research questions was a priority and required significant effort. One of the biggest challenges we encountered was merging datasets from different sources, which involved navigating inconsistencies in formats and missing connections between data tables. To address this, we applied database concepts, such as using primary and foreign keys, to establish meaningful links between tables. Furthermore, to align ourselves, we had to decide on which categories of products to keep and standardize other types of data such as the currencies used.
However, due to limitations in both time and resources, we had to rely on some random data generation to connect certain tables. Although this approach is not ideal, it was the most practical solution given the time and resource constraints we faced. This experience highlighted the importance of having the creative problem-solving skills required to work with fragmented data.