how to deal with mixed data ?

[xiaoguozhi]

great package on this subject! very nice job! here i have a problem.for example,I have a variable, such as ,dat<-data.frame(y=c(0,0,0,1,1,1,1,1,0,0,1,1,1,0,0,0,1,1,1,0),x=c(1,2,3,4,5,888,888,888,9,10,666,666,666,666,15,16,17,18,19,20)). In this case,i want regard '888' and '666'as special two class such as missing value have own woe, and i want to get two woe for '888' and '666' separately. other values are computed as usual. How to handle this type data. Thanks!

[ShichenXie]

Thanks. You can specify the breakpoints via option break_list in the function woebin. And you can get the optimal binning based on the dataset that excludes the two special values.

library(scorecard) library(data.table) dat<-data.table( y=c(0,0,0,1,1,1,1,1,0,0,1,1,1,0,0,0,1,1,1,0), x=c(1,2,3,4,5,888,888,888,9,10,666,666,666,666,15,16,17,18,19,20))

# get optimal breakpoints for rest dataset
bins <- woebin(dat[x != 666 & x != 888], "y")

# specify the breakpoints
bins2 <- woebin(dat, "y", breaks_list = list(x=c(16, 18, 666, 888)))
woebin_plot(bins2)

[xiaoguozhi]

Thanks. very detailed answer. In my case, i mean the value '666' and '888' is a categorical variable. so we should convert it as a factor before woebin.
wheile,the following code regard '666' and '888' as numerical variable.
bins2 <- woebin(dat, "y", breaks_list = list(x=c(16, 18, 666, 888)))
in my opinion, i can do following
library(scorecard)
library(data.table)
library(dplyr)
dat<-data.table( y=c(0,0,0,1,1,0,0,1,0,0,1,1,1,0,0,0,1,1,1,0), x=c(1,2,3,4,5,888,888,888,9,10,666,666,666,666,15,16,17,18,19,20))

special value

sp<-c(666,888)
#get the special data
dat_sp<-filter(dat, x %in% sp)
#get normal data
dat_nor<-filter(dat, !x %in% sp)

convert it to factor

dat_sp$x<-as.factor(dat_sp$x)
bins_sp <- woebin(dat_sp, "y")
woebin_plot(bins_sp)

bin for normal data

bins_nor <- woebin(dat_nor, "y")
woebin_plot(bins_nor)

and now the question is 1) how to combine these two plot in one plot. 2) how to combine these two woe for the variable in this case becase we can't do it by rbind function simply. if we have many such variable , how to get woe ? what I really warried is that we can't do bin for many variables automatically. for example,many functions in your package support batch process, obviously,if we bin for special value and normal value respectively,it destroies batch process.

[xiaoguozhi]

get optimal breakpoints for rest dataset

bins <- woebin(dat[x != 666 & x != 888], "y")
here, if we could really get the optimal breakponits once we omit some special samples?
and i realized that compute woe separately is wrong.

[ShichenXie]

It could be a problem if you have many variable that need to handle in this way.
I'll add this feature into woebin function. Thanks.

[xiaoguozhi]

Thanks again for your nice solution! Looking forward to your improved version.

[ShichenXie]

see the following example:

library(scorecard)

dat <- data.frame(y=c(0,0,0,1,1,1,1,1,0,0,1,1,1,0,0,0,1,1,1,0),
x=c(1,2,3,4,5,888,888,888,9,10,666,666,666,666,15,16,17,18,19,20))

#' specify two values as two class
bin = woebin(dat, "y", special_values = c(666,888))
#' specify two values as one class
bin2 = woebin(dat, "y", special_values = c("666%,%888"))