Create function to extract the leaf node assignments of an observation and output comparable observations

[dfsnow]

Overview / Context

Our office often gets asked "What properties did you look at to value my property?" or "What are my comparables?" There's not a clear answer to this question since our models effectively look at all available sales when training. However, there may be a way to extract pseudo-comparable observations by leveraging the structure of LightGBM's trees.

When LightGBM is training, it creates decision trees that split the continuous and categorical features into separate leaf nodes. For a single decision tree, it is likely that the observations that share the same terminal leaf nodes are very similar to each other, since they've ended up in the same split. We may be able to extract leaf node assignments and then back those assignments out into a set of comparables, or a similarity score.

The goal of this issue is to create a function that extracts either a set of comparables, a similarity index, or something similar. This will be part 1 of a longer project dedicated to making better tooling around comparables.

Task

Create a function to extract comparables or a score given a:

• Serialized LightGBM model object
• Parsnip workflow object
• Data frame of sales (training data)
• A vector of observation IDs that require comparables

The function should have a signature something like:

extract_comps <- function(object, workflow, new_data, obs_id) { 
   ...
}

It should return a nested list, where each nested item is a list of comparables for the input obs_ids.

Additional Requirements

• The function should include documentation and should be integrated into the larger package.
• Please submit a merge request when ready to close this issue.
• The function should take a vector input of IDs and quickly return an output. Imagine scaling this up to 1M+ IDs.
• Add as few dependencies as possible to the package DESCRIPTION. Ideally, add none.

[dfsnow]

Thanks for carrying this forward Billy! Here is some more context on some updates/next steps.

Speed optimization: based on a flame graph using profvis::profvis(), it seems like main way to speed up the extract comps function is to make the calculate_similarity_to() function go faster and work with vectorized inputs so that multiple original IDs can be fed in at once. The main issue here is figuring out the most efficient way to compare a vector of data (think all the leaf assignments of the base property) to a matrix row-wise (think all the other leaf assignments of each other property in the training data). Here are a few leads/documentation we’ve tried: data table join in place and apply() in data.table.

Weights update: I found that lightGBM calculates an “init_score” that it is trying to optimize for when it builds each subsequent tree. There might be an easier way to do it, but using lightgbm::set_field() and lightgbm::get_field(), I calculated the mean of the initial residuals of the “init_score” predictions – which perfectly matches the “init score” value that LightGBM prints when the model is being trained. By appending that as the first value of the errors, I used diff() to create a list of diff_in_errors and then calculated the new weights by dividing each diff by the sum of all diffs. This changed the weights pretty substantially – the first tree is worth 17% of the total similarity score now, which is up a lot from the previous weight calculation which was about 2%.

[dfsnow]

got a permissions error when trying to push, so this is the updated get weights function!

# get weights based on error 
get_weights_by = function(metric, tidymodels_model, training_data, outcome_col, num_trees) {
  # set metric in model params
  tidymodels_model$fit$params$metric = metric
  
  # create lgb.Dataset object for validation data
  training_data_lgbDataset <- lgb.Dataset(as.matrix(training_data), label = training_data[[outcome_col]])
  
  # create new model object with validation data
  model <- lgb.train(
    params = tidymodels_model$fit$params,
    data = training_data_lgbDataset,
    valids = list(test = training_data_lgbDataset),
    nrounds = num_trees)
  
  # get the initial error for base model before first tree
  # this NEEDS to be after the model is trained (or else it won't train correctly)
  set_field(training_data_lgbDataset, "init_score", as.matrix(training_data[[outcome_col]]))
  initial_predictions = get_field(training_data_lgbDataset, "init_score")
  init_score = mean(initial_predictions)
  
  # index into the errors list, and un-list so it is a flat/1dim list
  errors = unlist(model$record_evals$test[[metric]]$eval)
  errors = c(init_score, errors)
  diff_in_errors = diff(errors, 1, 1)
  
  # take proportion of diff in errors over total diff in errors from all trees
  weights = diff_in_errors / sum(diff_in_errors)
  
  return(weights)
}