This section provides an overview of the analysis and exploratory data analysis (EDA) performed on a dataset of diamond information. The analysis includes data preprocessing, visualization, and modelling preparation.
The aim of our project is to uncover relationships in the quality and features of gem diamonds and their impact on selling prices. By doing so, we aim to create tools for selecting different feature options based on specified values or predicting estimated pricing. This tool could then be used by sellers as a mechanism to entice customers.
The dataset used in this analysis was sourced from Kaggle, providing details on various attributes of diamonds, including their quality, characteristics, and pricing information.
https://www.kaggle.com/datasets/hrokrin/the-largest-diamond-dataset-currely-on-kaggle/data
Before running the code, ensure you have the following libraries and modules installed:
The analysis begins with loading the dataset and performing an initial inspection. The dataset is loaded using Pandas and the first few rows are displayed for a quick overview.
To ensure data quality, rows with zero values are removed from the dataset. This reduced the rows from 219,702 to 213,670.
We reviewed the counts by feature to assess the data distribution for machine learning.
We also removed rows where the count quantities of some features were extremely low and would be hard for a model to learn.
For example, cut_quality features of Good, Fair and Ideal had counts of 28, 5 and 1 respectively.
Other data rows removed were:
culet_condition features of Chipped (qty 18) and Abraded (qty 8), girdle_min features XTN and STN (quantities 290 and 24), and girdle_max features XTN and STN (quantities 33 and 12),fluor_intensity features of Unknown (qty 113) and Slight (qty 12) and finally polish features of Fair (qty 7) and Poor (qty 2)
This reduced our dataset to 213,134 rows.
Certain features are grouped or combined into new categories.
This approach helps in addressing low counts for certain quality values and making the data more manageable and meaningful for analysis and modelling. Feature logic was also taken into consideration to ensure the integrity of the feature was not negatively impacted.
Combinations are listed here: Clarity of I1, I2 and I3 wre all combined as they are all measurements of levels of inclusions (flaws) within the diamond. They had low counts (6717, 921, 84) Fluor_color features of Green, White and Orange had counts of 54, 42 and 9 respectively were combined into an “Other” value. Cut features of Cushion and Modified Cusion were combined into 1 value of "Cushion" (3914 and 515 counts) Symmetry features of Fair (319) and Poor (2) were combinned to an "Other" value Culet_size features of L, SL, EL and VL to a combined value of "L/SL/EL/VL" (a total of 77 rows) Fancy_color_dominant_color features of Purple, Gray, Blue, Chameleon, Red and Black were combined into an "Other" value with 181 total quantity. Fancy_color_secondary_color features of Red, Gray, Blue, and Violetwere combined into and "Other" value with 44 total quantity,
After binning the data, value counts are checked to verify the grouping process.
Categorical and numerical features are separated, and the target variable is defined.
Pearson correlation and heatmap are used to analyse the relationships between numerical features.
We wanted to understand how strongly each numerical feature is related to the target variable “total_sales_price” and Pearson correlation is suitable for identifying linear relationships.
We then ran a heatmap to visualize the Pearson correlation, making it easier to identify and interpret relationships between the features.
The strong positive correlation of 0.75 between carat weight and total sales price suggests that, in this dataset, higher carat weight diamonds tend to command higher prices.
This finding underscores the significance of carat weight as a key determinant of a diamond's value, potentially reflecting consumer preferences for larger, more substantial stones.
A Seaborn pair plot was created to visualize relationships among the features. Regression lines were added to the scatterplots for a better understanding of linear relationships.
We performed ANOVA as we wanted to investigate whether the different categories within the categorical features have a statistically significant impact on the target variable (total_sales_price).
This is a statistical method used to assess whether there are statistically significant differences between the means of two or more groups.
These results show that Cut, Cut quality, Color, Fancy color-dominant color and Fancy color intensity are significant categories in relation to the sales price (target variable).
Machine learning models are built to predict the total sales price of diamonds based on the features.
Three models are considered: Extra Trees Regressor, Random Forest Regressor with one-hot encoding, and Random Forest Regressor with label encoding.
The categorical features are encoded using one-hot encoding for use with the Extra Trees Regressor.
Created the ExtraTreesRegressor model and fit the data.
Get feature importances and display the top 12 features.
The categorical features are encoded using one-hot encoding for use with the Random Forest Regressor.
Created the RandomForestRegressor model and fit the data.
Get feature importances and display the top 12 features.
The categorical features are encoded using label encoding for use with the Random Forest Regressor.
Combined the Label Encoded categorical features with the numerical features.
Created the RandomForestRegressor model and fit the data.
Get feature importances and display top 12.
The significant features in 9.1 Extra Trees and 9.2 Random Forest Regressor methods are similar, based on the same hot encoding of categorical features. 9.3 Random Forest Regressor using label-encoded categorical features has a different set of important features.
Carat-weight had the highest importance by far in all 3 methods used, with the importance of other features dropping heavily in significance.
We considered the interpretability of the models and determined we are not interested in understanding how each category within a categorical variable impacts the total_sales_price (target variable) and therefore decided the model with label-encoding was more appropriate in this scenario.
The increase in dimensionality using one-hot ended data on an already large dataset would impact model complexity, training time and performance.
In the process of preparing data for modeling, we performed feature selection to determine which attributes have the most significant impact on our models. We created three different DataFrames for testing within the model training phase:
- Top 11 Features (Label Encoded Random Forest Regressor Model Results):
- We selected the top 11 features based on the results of the Label Encoded Random Forest Regressor model. These features are believed to be highly influential in predicting diamond prices.
- 6 Top Features (Random Forest Regressor Label Encoded Results):
- In this variant, we narrowed our focus to the top 6 features derived from the Random Forest Regressor Label Encoded results. We excluded measurement-related features to assess their impact on model training.
- The "4 C's" DataFrame:
- This DataFrame comprises four essential attributes in the diamond classification known as the "4 C's" – Cut, Color, Clarity, and Carat weight. These attributes are often fundamental factors in diamond purchasing decisions.
These DataFrames represent different sets of features used in our analysis. The final dataset, including label-encoded categorical data and numerical data, is saved and exported to a CSV file for further modeling.
This project focuses on training a machine learning model for predicting diamond sales prices using two different datasets. The first dataset includes the top 11 features, achieving an accuracy of 0.87, while the second dataset comprises the top 4 features, resulting in an accuracy of 0.82.
- Setting up top 4 Features and Target
- We start by setting up the features and target variable for the diamond price prediction. The features include various attributes such as cut, color, clarity, and carat weight, while the target is the price of the diamond.
- Setting up top 11 Features and Target
- We start by setting up the features and target variable for the diamond price prediction. The features include various attributes such as cut, color, clarity, depth percent, table percent, meas length, meas width, meas depth, fancy color, fancy color intensity and carat weight, while the target is the price of the diamond.
We create pipelines that standardise the data and include four different regression models: Linear Regression, Decision Tree, Random Forest, K-Nearest Neighbors, and the model with the best cross-validation score.
Train all selected models using the training data, each using either the top 11 or top 4 features dataset.
We calculate the mean of cross-validation scores for the models using the negative root mean square error as the evaluation metric. Based on the cross-validation scores, we identify the model with the best performance. This model will be selected for further evaluation and testing.
We also utilized a deep learning neural network to demonstrate model optimization, training it over 100 epochs and achieving an R2 score of 0.82, we ultimately decided with Random Forest regression model.
The decision was driven by the need for a more interpretable and transparent approach. Although the neural network displayed strong predictive capabilities, its complexity made it challenging to provide meaningful insights into the model's decision-making process. In contrast, the Random Forest regression model offers transparency and interpretability, allowing us to gain a clearer understanding of the factors influencing predictions.
By choosing the Random Forest regression model, we aim to strike a balance between model performance and interpretability, ensuring that we can provide actionable insights based on the features that most impact our predictions. This decision aligns with our goal of not only achieving accurate predictions but also comprehending the underlying drivers of those predictions, ultimately leading to more informed decision-making.
Random Forest appears to be the model with the best scoring on negative root mean square error. We proceed to test this model on a separate test set and evaluate its performance using various parameters and metrics.
This part of the process is focused on Web Page development, pass data from a web page, run it on a model and bring the result back to the page.
-
HTML is used to create the basic structure of the web page.
-
CSS is used for the size, style and position of different elementson the web page.
-
JavaScript is used for interactivity between different sections and elements of the web page.
-
HTML form element dropdown is used to collect user input.
-
We needed a Web Server to handle HTTP requests and responses, so we used Python and Flask for this purpose. We also created an API endpoint on the server to which the web page can send data. This API endpoint processes the data and return the result.
-
We used a library called "Pickle" to load the pre-trained machine learning model on the server from local disk and the data is processed and transformed into required format before feeding it to the model. The model then process the data and make predictions.
-
We used JavaScript to send the data to the API endpoint from the web page and after the data is processed and the pridiction is made, the server formats the response and send the result back to the web page using HTTP response. JavaScript on the web page will receive the result and update the DOM (Document Object Model) to display the prediction on the page.
Created and submitted as Group Project for Monash University Data Analytics Boot Camp (November 2023).
Data collated, cleaned and code written by:
• Jancel Adiong • Sagar Bora • Helena Chen • Caroline Grant
-
Diamond data:
https://www.kaggle.com/datasets/hrokrin/the-largest-diamond-dataset-currely-on-kaggle/data
-
Diamond information