11. Lessons Learned

Challenges faced and how they were overcome

Subgroup A

Customer Segmentation model

One of the primary challenges in developing our customer segmentation model was the limited coverage of segmentation techniques in our lectures. While our course included a general overview of machine learning, the emphasis was largely on machine-based models rather than specifically on segmentation. The only machine learning model that seemed to be suitable for segmentation was unsupervised learning that was in the summary slide, with minimal guidance on how to apply these models for customer segmentation. This gap meant that I needed to independently research and study various unsupervised learning techniques to understand and implement an effective segmentation approach.

After testing several times with different unsupervised learning methods, I identified an optimal number of clusters as k=64, based on standard unsupervised clustering evaluation metrics. However, this result presented a second major challenge: from a business perspective, such a high number of clusters was impractical and extremely challenging to interpret. This outcome highlighted the need for alternative segmentation approaches that align better with business goals, which had not been covered in our lectures.

To overcome these challenges, I sought advice from our teaching assistant (TA), Hera, in a consultation session. Hera introduced me to a human knowledge-based approach for customer segmentation, where clusters could be designed around human-defined behavioral attributes. This approach provided a more practical, interpretable solution for our project and was highly effective in creating meaningful customer segments based on business insights.

Insights gained during the project

Reflections on the team's approach and process

In this project, sourcing datasets that could effectively address our research questions was a priority and required significant effort. One of the biggest challenges we encountered was merging datasets from different sources, which involved navigating inconsistencies in formats and missing connections between data tables. To address this, we applied database concepts, such as using primary and foreign keys, to establish meaningful links between tables.

However, due to limitations in both time and resources, we had to rely on some random data generation to connect certain tables. Although this approach is not ideal, it was the most practical solution given the time and resource constraints we faced. This experience highlighted the importance of having the creative problem-solving skills required to work with fragmented data.