Data Preparation and Exploration Project

Overview

This project focuses on understanding the utility of data preparation and exploration techniques using a dataset chosen from a predefined list. By analyzing and preparing the dataset, we aim to extract meaningful insights and prepare the data for further analysis or modeling. The project also emphasizes the importance of data cleaning, feature engineering, and visualization to understand and interpret the data effectively.

---

Datasets

The following is the dataset we chose to use:

Loan Approval Dataset
Dataset Link

---

Tasks and Requirements

Dataset Analysis and Preparation

1. Data Exploration:

   • Display the first and last 12 rows of the dataset.
   • Identify the total number of rows and columns.
   • List all column names and their data types.
   • Print the name of the first column.
   • Generate a summary of the dataset, including non-null counts and data types.
   • Choose a categorical attribute and display its distinct values.
   • Identify the most frequently occurring value in the categorical attribute.
   • Calculate and present the mean, median, standard deviation, and percentiles for a numerical attribute.

2. Data Preparation:

   • Filter rows based on a specific condition of your choice.
   • Count records where an attribute starts with a specific letter.
   • Identify and remove duplicate rows.
   • Convert a numerical column's data type (e.g., integer to string).
   • Group data by two categorical features and analyze the results.
   • Check for missing values and replace them with the median or mode as appropriate.
   • Divide a numerical column into 5 equal-width bins and count the records in each bin.
   • Identify and print the row corresponding to the maximum value of a numerical feature.
   • Create a boxplot, histogram, and scatterplot for selected attributes and justify their relevance.
   • Normalize numerical attributes using StandardScaler.
   • Apply Principal Component Analysis (PCA) to reduce the dataset to two components and visualize the data before and after PCA.
   • Analyze correlations between numerical features using a heatmap.

Practical Analytical Questions

• Calculate and display the correlation matrix to identify relevant features for classification.
• Find the class distribution of a selected categorical feature and analyze the results.
• Perform feature engineering by creating new features from existing ones and explain their significance.

---