[EDA] Feature Associations

Studying association of data set features

1. Univariate
   1. Histogram
2. Bivariate
   1. Correlation and scatter plots (pairs) for quantitative-vs-quantitative pairs
   2. Boxplot and barcode plots for quantitative-vs-categorical pairs

Correlation and Redundancy

1. Highest value is 1.0 which means 2 variables are perfectly linearly related
   1. Shows that 2 features are redundant and we only need to keep 1
2. Less correlation will be more interesting for a machine learning model to learn from

R Cheat Sheet

R^2

calculate_r_2 <- function(actual, prediction) {
    return (1 - (sum((actual-prediction)^2)/sum((actual-mean(actual))^2)))
}

calculate_r_2_for_feature <- function(data, feature) {
    n <- nrow(data)
    
    train_index <- sample(seq_len(n), size = 0.8*n)

    train <- data[train_index,]
    test <- data[-train_index,]
    
    this_formula = paste(feature,"~.")
    fit <- rpart(data=train, formula=as.formula(this_formula))

    y_test <- as.vector(test[[feature]])
    test[feature] <- NULL
    predictions <- predict(fit, test)
    return (calculate_r_2(y_test, predictions))
}

Pearson Correlation Plot

library(ggplot2)
library(reshape2)
ggplot(data = melt(cor(df), na.rm = T), aes(Var2, Var1, fill = value))+
 geom_tile(color = "white")+
 scale_fill_gradient2(low = "blue", high = "red", mid = "white", 
   midpoint = 0, limit = c(-1,1), space = "Lab", 
   name="Pearson\nCorrelation") +
  theme_minimal()+ 
 theme(axis.text.x = element_text(angle = 45, vjust = 1, 
    size = 12, hjust = 1))+
 coord_fixed()