preprocessing.qmd

---
aliases:
  - "/preprocessing.html"
---

# Preprocessing {#sec-preprocessing}

{{< include ../../common/_setup.qmd >}}

`r chapter = "Preprocessing"`
`r authors(chapter)`

@sec-pipelines and @sec-pipelines-nonseq provided a technical introduction to `r mlr3pipelines`, this chapter will now demonstrate how to use those pipelines to tackle common problems when `r index('preprocessing')` data for ML, including `r index('factor encoding')`, `r index('imputation')` of missing values, feature and target transformations, and functional `r index('feature extraction')`.
Feature selection, an important preprocessing method, is covered in @sec-feature-selection.

In this book, preprocessing refers to everything that happens with *data* before it is used to fit a model, while `r index('postprocessing')` encompasses everything that occurs with *predictions* after the model is fitted.

`r index('Data cleaning', aside = TRUE)`\index{exploratory data analysis|see{data cleaning}} is an important part of preprocessing that involves the removal of errors, noise, and redundancy in the data; we only consider data cleaning very briefly as it is usually performed outside of `mlr3` on the raw dataset.

Another aspect of preprocessing is `r index('feature engineering', aside = TRUE)`, which covers all other transformations of data before it is fed to the machine learning model, including the creation of features from possibly unstructured data, such as written text, sequences or images.
The goal of feature engineering is to enable the data to be handled by a given learner, and/or to further improve predictive performance.
It is important to note that feature engineering helps mostly for simpler algorithms, while highly complex models usually gain less from it and require little data preparation to be trained.
Common difficulties in data that can be solved with feature engineering include features with skewed distributions, high-cardinality categorical features, missing observations, high dimensionality and imbalanced classes in classification tasks.
Deep learning has shown promising results in automating feature engineering, however, its effectiveness depends on the complexity and nature of the data being processed, as well as the specific problem being addressed.
Typically it can work well with natural language processing and computer vision problems, while for standard tabular data, tree-based ensembles such as a random forest or gradient boosting are often still superior (and easier to handle). However, tabular deep learning approaches are currently catching up quickly.
Hence, manual feature engineering is still often required but with `mlr3pipelines`, which can simplify the process as much as possible.

As we work through this chapter we will use an adapted version of the Ames housing data [@de2011ames].
We changed the data slightly and introduced some additional (artificial) problems to showcase as many aspects of preprocessing as possible on a single dataset.
The modified version is shipped with `r ref_pkg("mlr3data")` and the code to recreate this version of the data from the original raw data can be found at `r link("https://github.com/mlr-org/mlr3data/")` in the directory `data-raw`.
This original dataset was collected as an alternative to the Boston Housing data and is commonly used to demonstrate feature engineering in ML.
Raw and processed versions of the data can be directly loaded from the `r ref_pkg("AmesHousing")` package.
The dataset includes 2,930 residential properties (rows) situated in Ames, Iowa, sold between 2006 and 2010.
It contains 81 features about various aspects of the property, the size and shape of the lot, and information about its condition and quality.
The prediction target is the sale price in USD, hence it is a regression task.

```{r preprocessing-001, message=FALSE}
ames = mlr3data::ames_housing
```

## Data Cleaning

As a first step, we explore the data and look for simple problems such as constant or duplicated features.
This can be done quite efficiently with a package like `r ref_pkg("DataExplorer")` or `r ref_pkg("skimr")` which can be used to create a large number of informative plots.

Below we summarize the most important findings for data cleaning, but we only consider this aspect in a cursory manner:

```{r preprocessing-002, message=FALSE}
# 1. `Misc_Feature_2` is a factor with only a single level `Othr`.
summary(ames$Misc_Feature_2)
# 2. `Condition_2` and `Condition_3` are identical.
identical(ames$Condition_2, ames$Condition_3)
# 3. `Lot_Area` and `Lot_Area_m2` are same data on different scales
cor(ames$Lot_Area, ames$Lot_Area_m2)
```

For all three problems, simply removing the problematic features (or feature in a pair) might be the best course of action.

```{r preprocessing-003, message=FALSE}
to_remove = c("Lot_Area_m2", "Condition_3", "Misc_Feature_2")
```

Other typical problems that should be checked are:

1. ID columns, i.e., columns that are unique for every observation should be removed or tagged.
2. `NA`s not correctly encoded, e.g. as `"NA"` or `""`
3. Semantic errors in the data, e.g., negative `Lot_Area`
4. Numeric features encoded as categorical for learners that can not handle such features.

Before we continue with feature engineering we will create a task, measure, and resampling strategy to use throughout the chapter.

```{r preprocessing-004, message=FALSE}
tsk_ames = as_task_regr(ames, target = "Sale_Price", id = "ames")
# remove problematic features
tsk_ames$select(setdiff(tsk_ames$feature_names, to_remove))

msr_mae = msr("regr.mae")
rsmp_cv3 = rsmp("cv", folds = 3)
rsmp_cv3$instantiate(tsk_ames)
```

Lastly, we run a very simple experiment to verify our setup works as expected with a simple featureless baseline, note below we set `robust = TRUE` to always predict the *median* sale price as opposed to the *mean*.

```{r preprocessing-005, message=FALSE}
lrn_baseline = lrn("regr.featureless", robust = TRUE)
lrn_baseline$id = "Baseline"
rr_baseline = resample(tsk_ames, lrn_baseline, rsmp_cv3)
rr_baseline$aggregate(msr_mae)
```

## Factor Encoding

Many machine learning algorithm implementations, such as XGBoost [@chen2016xgboost], cannot handle categorical data and so categorical features must be encoded\index{encoding} into numerical variables.

```{r preprocessing-006, message=FALSE, error=TRUE}
lrn_xgb = lrn("regr.xgboost", nrounds = 100)
lrn_xgb$train(tsk_ames)
```

Categorical features can be grouped by their cardinality, which refers to the number of levels they contain: binary features (two levels), low-cardinality features, and high-cardinality features; there is no universal threshold for when a feature should be considered high-cardinality and this threshold can even be tuned.
For now, we will consider high-cardinality to be features with more than 10 levels:

```{r preprocessing-007}
names(which(lengths(tsk_ames$levels()) > 10))
```

Binary features can be trivially encoded by setting one of the feature levels to `1` and the other to `0`.

```{r preprocessing-008}
names(which(lengths(tsk_ames$levels()) == 2))
```

Low-cardinality features can be handled by `r index('one-hot encoding', 'one-hot', parent = 'encoding', aside = TRUE)`.
One-hot encoding is a process of converting categorical features into a binary representation, where each possible category is represented as a separate binary feature.
Theoretically, it is sufficient to create one less binary feature than levels, as setting all binary features to zero is also a valid representation.
This is typically called dummy\index{dummy encoding|see{encoding, treatment}} or treatment encoding\index{encoding!treatment} and is required if the learner is a generalized linear model (GLM) or additive model (GAM)\index{generalized linear model}.

Some learners support handling categorical features but may still crash for high-cardinality features if they internally apply encodings that are only suitable for low-cardinality features, such as one-hot encoding.
Impact encoding [@MicciBarreca2001] is a good approach for handling high-cardinality features.
`r index('Impact encoding', 'impact', parent = 'encoding', aside = TRUE)` converts categorical features into numeric values.
The idea behind impact encoding is to use the target feature to create a mapping between the categorical feature and a numerical value that reflects its importance in predicting the target feature.
Impact encoding involves the following steps:

1. Group the target variable by the categorical feature.
2. Compute the mean of the target variable for each group.
3. Compute the global mean of the target variable.
4. Compute the impact score for each group as the difference between the mean of the target variable for the group and the global mean of the target variable.
5. Replace the categorical feature with the impact scores.

Impact encoding preserves the information of the categorical feature while also creating a numerical representation that reflects its importance in predicting the target.
Compared to one-hot encoding, the main advantage is that only a single numeric feature is created regardless of the number of levels of the categorical features, hence it is especially useful for high-cardinality features.
As information from the target is used to compute the impact scores, the encoding process must be embedded in cross-validation to avoid leakage between training and testing data (@sec-performance).

As well as encoding features, other basic preprocessing steps for categorical features include removing constant features (which only have one level and may have been removed as part of data cleaning), and collapsing levels that occur very rarely.
These types of problems can occur as artifacts of resampling as the dataset size is further reduced.
Stratification on such features would be an alternative way to mitigate this (@sec-strat-group).

In the code below we use `po("removeconstants")` to remove features with only one level, `po("collapsefactors")` to collapse levels that occur less than 1% of the time in the data, `po("encodeimpact")` to impact-encode high-cardinality features, `po("encode", method = "one-hot")` to one-hot encode low-cardinality features, and finally `po("encode", method = "treatment")` to treatment encode binary features.

```{r preprocessing-009, message=FALSE}
factor_pipeline =
    po("removeconstants") %>>%
    po("collapsefactors", no_collapse_above_prevalence = 0.01) %>>%
    po("encodeimpact",
        affect_columns = selector_cardinality_greater_than(10),
        id = "high_card_enc") %>>%
    po("encode", method = "one-hot",
        affect_columns = selector_cardinality_greater_than(2),
        id = "low_card_enc") %>>%
    po("encode", method = "treatment",
        affect_columns = selector_type("factor"), id = "binary_enc")
```

The order in which operations are performed matters here: `po("encodeimpact")` converts high-cardinality `factor` type features into `numeric` features, so these will not be affected by the `po("encode")` operators that come afterwards.
Therefore, the one-hot encoding PipeOp does not need to specify *not* to affect high-cardinality features.
Likewise, once the treatment encoding PipeOp sees the data, all non-binary `factor` features have been converted, so it will only affect binary factors by default.

Now we can apply this pipeline to our xgboost model to use it in a benchmark experiment; we also compare a simpler pipeline that only uses one-hot encoding to demonstrate performance differences resulting from different strategies.

```{r preprocessing-010, message=FALSE}
glrn_xgb_impact = as_learner(factor_pipeline %>>% lrn_xgb)
glrn_xgb_impact$id = "XGB_enc_impact"

glrn_xgb_one_hot = as_learner(po("encode") %>>% lrn_xgb)
glrn_xgb_one_hot$id = "XGB_enc_onehot"

bmr = benchmark(benchmark_grid(tsk_ames,
  c(lrn_baseline, glrn_xgb_impact, glrn_xgb_one_hot), rsmp_cv3))
bmr$aggregate(measure = msr_mae)[, .(learner_id, regr.mae)]
```

In this small experiment, we see that the difference between the extended factor encoding pipeline and the simpler one-hot encoding strategy pipeline is only very small.
If you are interested in learning more about different encoding strategies, including a benchmark study comparing them, we recommend @pargent2022regularized.

## Missing Values {#sec-preprocessing-missing}

A common problem in real-world data is `r index('missing values', 'missing data')` in features.
In the Ames dataset, several variables have at least one missing data point:

```{r preprocessing-011}
# print first five with missing data
names(which(tsk_ames$missings() > 0))[1:5]
```

Many learners cannot handle missing values automatically (e.g., `lrn("regr.ranger")` and `lrn("regr.lm")`) and others may be able to handle missing values but may use simple methods that are not ideal (e.g., just omitting rows with missing data).

The simplest `r index('data imputation', aside = TRUE)` method is to replace missing values by the feature's mean (`po("imputemean")`) (@fig-imputation), median (`po("imputemedian")`), or mode (`po("imputemode")`).
Alternatively, one can impute by sampling from the empirical distribution of the feature, for example a histogram (`po("imputehist")`).
Instead of guessing at what a missing feature might be, missing values could instead be replaced by a new level, for example, called `.MISSING` (`po("imputeoor")`).
For numeric features, @ding2010investigation show that for binary classification and tree-based models, encoding missing values out-of-range (OOR), e.g. a constant value above the largest observed value, is a reasonable approach.

```{r preprocessing-012, echo = FALSE, out.width = "60%"}
#| label: fig-imputation
#| fig-cap: Mean imputation of missing values using observed values.
#| fig-alt: "On the left is a vector of numbers in a column, (1.3, NA, 1.5, NA). The non-NA numbers have arrows pointing to (1.3+1.5)/2, which then has an arrow pointing to a vector of numbers in a column on the right but now (1.3, 1.4, 1.5, 1.4) with '1.4' in red to highlight they were imputed with the mean."
include_multi_graphics("mlr3book_figures-13")
```

It is often important for predictive tasks that you keep track of missing data as it is common for missing data to be informative in itself.
To preserve the information about which data was missing, imputation should be tracked by adding binary indicator features (one for each imputed feature) that are `1` if the feature was missing for an observation and `0` if it was present (`po("missind")`).
It is important to note that recording this information will not prevent problems in model interpretation on its own.
As a real-world example, medical data are typically collected more extensively for White communities than for racially minoritized communities.
Imputing data from minoritized communities would at best mask this data bias, and at worst would make the data bias even worse by making vastly inaccurate assumptions (see @sec-fairness for data bias and algorithmic fairness).

In the code below we create a pipeline from the `r ref("PipeOp")`s listed above as well as making use of `po("featureunion")` to combine multiple `PipeOp`s acting on the `"integer"` columns.

```{r preprocessing-013, eval = FALSE}
impute_hist = list(
      po("missind", type = "integer",
          affect_columns = selector_type("integer")
      ),
      po("imputehist", affect_columns = selector_type("integer"))
    ) %>>%
    po("featureunion") %>>%
    po("imputeoor", affect_columns = selector_type("factor"))

impute_hist$plot(horizontal = TRUE)
```

```{r preprocessing-014, fig.width = 8, echo = FALSE}
#| label: fig-impute
#| fig-cap: Pipeline to impute missing values of numeric features by histogram with binary indicators and missings in categoricals out-of-range with a new level.
#| fig-alt: "Flow diagram shows '<INPUT>' with arrows to 'missind' and 'imputehist', which both have arrows to 'featureunion', which has an arrow to 'imputeoor' that has an arrow to '<OUTPUT'>."
impute_hist = list(
      po("missind", type = "integer",
          affect_columns = selector_type("integer")
      ),
      po("imputehist", affect_columns = selector_type("integer"))
    ) %>>%
    po("featureunion") %>>%
    po("imputeoor", affect_columns = selector_type("factor"))

fig = magick::image_graph(width = 1500, height = 1000, res = 100, pointsize = 24)
impute_hist$plot(horizontal = TRUE)
invisible(dev.off())
magick::image_trim(fig)
```

Using this pipeline we can now run experiments with `lrn("regr.ranger")`, which cannot handle missing data; we also compare a simpler pipeline that only uses OOR imputation to demonstrate performance differences resulting from different strategies.

```{r preprocessing-015}
glrn_rf_impute_hist = as_learner(impute_hist %>>% lrn("regr.ranger"))
glrn_rf_impute_hist$id = "RF_imp_Hist"

glrn_rf_impute_oor = as_learner(po("imputeoor") %>>% lrn("regr.ranger"))
glrn_rf_impute_oor$id = "RF_imp_OOR"

design = benchmark_grid(tsk_ames,
  c(glrn_rf_impute_hist, glrn_rf_impute_oor), rsmp_cv3)
bmr_new = benchmark(design)
bmr$combine(bmr_new)
bmr$aggregate(measure = msr_mae)[, .(learner_id, regr.mae)]
```

Similarly to encoding, we see limited differences in performance between the different imputation strategies.
This is expected here and confirms the findings of @ding2010investigation -- out-of-range imputation is a simple yet effective imputation for tree-based methods.

Many more advanced imputation strategies exist, including model-based imputation where machine learning models are used to predict missing values, and multiple imputation where data is repeatedly resampled and imputed in each sample (e.g., by mean imputation) to attain more robust estimates.
However, these more advanced techniques rarely improve the models predictive performance substantially and the simple imputation techniques introduced above are usually sufficient [@Poulos2018].
Nevertheless, these methods are still important, as finding imputations that fit well to the distribution of the observed values allows a model to be fitted that can be interpreted and analyzed in a second step.

## Pipeline Robustify {#sec-prepro-robustify}

`mlr3pipelines` offers a simple and reusable pipeline for (among other things) `r index('imputation')` and factor `r index('encoding')` called `r index('ppl("robustify")', aside = TRUE, code = TRUE)`, which includes sensible defaults that can be used most of the time when encoding or imputing data.
The pipeline includes the following `r ref("PipeOp")`s (some are applied multiple times and most use selectors):

1. `po("removeconstants")` -- Constant features are removed.
2. `po("colapply")` -- Character and ordinal features are encoded as categorical, and date/time features are encoded as numeric.
3. `po("imputehist")` -- Numeric features are imputed by histogram sampling.
4. `po("imputesample")` -- Logical features are imputed by sampling from the empirical distribution -- this only affects the `$predict()`-step.
5. `po("missind")` -- Missing data indicators are added for imputed numeric and logical variables.
6. `po("imputeoor")` -- Missing values of categorical features are encoded with a new level.
7. `po("fixfactors")` -- Fixes levels of categorical features such that the same levels are present during prediction and training (which may involve dropping empty factor levels).
8. `po("imputesample")` -- Missing values in categorical features introduced from dropping levels in the previous step are imputed by sampling from the empirical distributions.
9. `po("collapsefactors")` -- Categorical features levels are collapsed (starting from the rarest factors in the training data) until there are less than a certan number of levels, controlled by the `max_cardinality` argument (with a conservative default of `1000`).
10. `po("encode")` -- Categorical features are one-hot encoded.
11. `po("removeconstants")` -- Constant features that might have been created in the previous steps are removed.

`ppl("robustify")` has optional arguments `task` and `learner`.
If these are provided, then the resulting pipeline will be set up to handle the given task and learner specifically, for example, it will not impute missing values if the learner has the `"missings"` property, or if there are no missing values in the task to begin with.
By default, when `task` and `learner` are not provided, the graph is set up to be defensive: it imputes all missing values and converts all feature types to numerics.

Linear regression is a simple model that cannot handle most problems that we may face when processing data, but with the `ppl("robustify")` we can now include it in our experiment:

```{r preprocessing-016, warning = FALSE}
glrn_lm_robust = as_learner(ppl("robustify") %>>% lrn("regr.lm"))
glrn_lm_robust$id = "lm_robust"

bmr_new = benchmark(benchmark_grid(tsk_ames, glrn_lm_robust,  rsmp_cv3))
bmr$combine(bmr_new)
bmr$aggregate(measure = msr_mae)[, .(learner_id, regr.mae)]
```

Robustifying the linear regression results in a model that vastly outperforms the featureless baseline and is competitive when compared to more complex machine learning models.

## Transforming Features and Targets {#sec-prepro-scale}

Simple transformations of features and the target can be beneficial (and sometimes essential) for certain learners.
In particular, log transformation of the target can help in making the distribution more symmetrical and can help reduce the impact of outliers.
Similarly, log transformation of skewed features can help to reduce the influence of outliers.
In @fig-sale we plot the distribution of the target in the `ames` dataset and then the log-transformed target, we can see how simply taking the log of the variable results in a distribution that is much more symmetrical and with fewer outliers.

```{r preprocessing-017, message=FALSE}
#| output: false
#| cache: false
library(patchwork)

# copy ames data
log_ames = copy(ames)
# log transform target
log_ames[, logSalePrice := log(Sale_Price)]
# plot
autoplot(as_task_regr(log_ames, target = "Sale_Price")) +
  autoplot(as_task_regr(log_ames, target = "logSalePrice"))
```

```{r preprocessing-018}
#| label: fig-sale
#| fig-cap: Distribution of house sales prices (in USD) in the ames dataset before (left) and after (right) log transformation. Before transformation there is a skewed distribution of prices towards cheaper properties with a few outliers of very expensive properties. After transformation the distribution is much more symmetrical with the majority of points evenly spread around the same range.
#| fig-alt: Two boxplots. Left plot shows house prices up to $600,000, the majority of prices are between roughly $100,000-$200,000. Right plot shows log house prices primarily around 12 with an even range between 11 and 13 and a few outliers on both sides.
#| echo: false
#| warning: false
#| message: false
plt = ggplot2::last_plot()
plt$layers[[1]]$aes_params$fill = NULL
plt$patches$plots[[1]]$layers[[1]]$aes_params$fill = NULL
print(plt)
```

Normalization of features may also be necessary to ensure features with a larger scale do not have a higher impact, which is especially important for distance-based methods such as `r index('k-nearest neighbors')` models or regularized parametric models such as Lasso or Elastic net.
Many models internally scale the data if required by the algorithm so most of the time we do not need to manually do this in preprocessing, though if this is required then `po("scale")` can be used to center and scale numeric features.

Any transformations applied to the target during training must be inverted during model prediction to ensure predictions are made on the correct scale.
By example, say we are interested in log transforming the target, then we would take the following steps:

```{r preprocessing-019}
df = data.table(x = runif(5), y = runif(5, 10, 20))
df
# 1. log transform the target
df[, y := log(y)]
df$y
# 2. make linear regression predictions
#    predictions on the log-transformed scale
yhat = predict(lm(y ~ x, df), df)
yhat
# 3. transform to correct scale with inverse of log function
#    predictions on the original scale
exp(yhat)
```

In this simple experiment, we could manually transform and invert the target, however, this is much more complex when dealing with resampling and benchmarking experiments and so the pipeline `ppl("targettrafo")` will do this heavy lifting for you.
The pipeline includes a parameter `targetmutate.trafo` for the transformation to be applied during training to the target, as well as `targetmutate.inverter` for the transformation to be applied to invert the original transformation during prediction.
So now let us consider the log transformation by adding this pipeline to our robust linear regression model:

```{r preprocessing-020, warning=FALSE}
glrn_log_lm_robust = as_learner(ppl("targettrafo",
  graph = glrn_lm_robust,
  targetmutate.trafo = function(x) log(x),
  targetmutate.inverter = function(x) list(response = exp(x$response))))
glrn_log_lm_robust$id = "lm_robust_logtrafo"

bmr_new = benchmark(benchmark_grid(tsk_ames, glrn_log_lm_robust,
  rsmp_cv3))
bmr$combine(bmr_new)
bmr$aggregate(measure = msr_mae)[, .(learner_id, regr.mae)]
```

With the target transformation and the `ppl("robustify")`, the simple linear regression now appears to be the best-performing model.

## Functional Feature Extraction

As a final step of data preprocessing, we will look at `r index('feature extraction')` from functional features.
In @sec-feature-selection we look at automated `r index('feature selection')` and how automated approaches with filters and wrappers can be used to reduce a dataset to an optimized set of features.
Functional feature extraction differs from this process as we are now interested in features that are dependent on one another and together may provide useful information but not individually.
@fig-functional-features visualizes the difference between regular and functional features.

```{r preprocessing-021, echo = FALSE}
#| label: fig-functional-features
#| fig-cap: Variables x1,x2,x3 are regular features, variables xt1,...,xt365 are functional features that could be plotted to identify important properties.
#| fig-alt: On the left is a table with columns 'x1,x2,x3,xt1,xt2,...,xt365'. Below the first three columns is the label 'Regular Features', below the others is the label 'Functional Features, e.g. days in year'. The table has a bidirectional arrow to a line graph that indicates plotting of one row of functional features.
include_multi_graphics("mlr3book_figures-14")
```

As a concrete example, consider the power consumption of kitchen appliances in houses in the Ames dataset.

```{r preprocessing-022, message=FALSE, warning=FALSE}
energy_data = mlr3data::energy_usage
```

In this dataset, each row represents one house and each feature is the total power consumption from kitchen appliances at a given time [@bagnall2017great].
The consumption is measured in two-minute intervals, resulting in 720 features.

```{r preprocessing-023, message=FALSE, warning=FALSE}
#| label: fig-energy
#| fig-cap: Energy consumption of one example house in a day, recorded in two-minute intervals.
#| fig-alt: Line plot with '2-Minute Interval' on axis ranging from 1 to 720 and 'Power Consumption' on y-axis ranging from 0 to 20. There are spikes at around (200, 20), (300, 20), and then some consistently raised usage between (500-700, 3).
library(ggplot2)
ggplot(data.frame(y = as.numeric(energy_data[1, ])),
    aes(y = y, x = 1:720)) +
  geom_line() + theme_minimal() +
  labs(x = "2-Minute Interval", y = "Power Consumption")
```

Adding these 720 features to our full dataset is a bad idea as each individual feature does not provide meaningful information, similarly, we cannot automate selection of the best feature subset for the same reason.
Instead, we can *extract* information about the curves to gain insights into the kitchen's overall energy usage.
For example, we could extract the maximum used wattage, overall used wattage, number of peaks, and other similar features.

To extract features we will write our own `r ref("PipeOp")` that inherits from `r ref("PipeOpTaskPreprocSimple")`.
To do this we add a private method called `.transform_dt` that hardcodes the operations in our task.
In this example, we select the functional features (which all start with "att"), extract the mean, minimum, maximum, and variance of the power consumption, and then remove the functional features.
To read more about building custom `PipeOp`s, open the corresponding vignette by running `vignette("extending", package = "mlr3pipelines")` in R.

```{r preprocessing-024}
PipeOpFuncExtract = R6::R6Class("PipeOpFuncExtract",
  inherit = mlr3pipelines::PipeOpTaskPreprocSimple,
  private = list(
    .transform_dt = function(dt, levels) {
        ffeat_names = paste0("att", 1:720)
        ffeats = dt[, ..ffeat_names]
        dt[, energy_means := apply(ffeats, 1, mean)]
        dt[, energy_mins := apply(ffeats, 1, min)]
        dt[, energy_maxs := apply(ffeats, 1, max)]
        dt[, energy_vars := apply(ffeats, 1, var)]
        dt[, (ffeat_names) := NULL]
        dt
    }
  )
)
```

Before using this in an experiment we first test that the `PipeOp` works as expected.

```{r preprocessing-025}
tsk_ames_ext = cbind(ames, energy_data)
tsk_ames_ext = as_task_regr(tsk_ames_ext, "Sale_Price", "ames_ext")
# remove the redundant variables identified at the start of this chapter
tsk_ames_ext$select(setdiff(tsk_ames_ext$feature_names, to_remove))

func_extractor = PipeOpFuncExtract$new("energy_extract")
tsk_ames_ext = func_extractor$train(list(tsk_ames_ext))[[1]]
tsk_ames_ext$data(1,
  c("energy_means", "energy_mins", "energy_maxs", "energy_vars"))
```

These outputs look sensible compared to @fig-energy so we can now run our final benchmark experiment using feature extraction.
We do not need to add the `PipeOp` to each learner as we can apply it once (as above) before any model training by applying it to all available data.

```{r preprocessing-026, warning=FALSE, R.options = list(datatable.print.nrows = 13, datatable.print.class = FALSE, datatable.print.keys = FALSE, datatable.print.trunc.cols = TRUE)}
learners = list(lrn_baseline, lrn("regr.rpart"), glrn_xgb_impact,
    glrn_rf_impute_oor, glrn_lm_robust, glrn_log_lm_robust)

bmr_final = benchmark(benchmark_grid(c(tsk_ames_ext, tsk_ames), learners,
  rsmp_cv3))

perf = bmr_final$aggregate(measure = msr_mae)
perf[order(learner_id, task_id), .(task_id, learner_id, regr.mae)]
```

The final results indicate that adding these extracted features improved the performance of all models (except the featureless baseline).

In this example, we could have just applied the transformations to the dataset directly and not used a `PipeOp`.
However, the advantage of using the `PipeOp` is that we could have chained it to a subset of learners to prevent a blow-up of experiments in the benchmark experiment.

## Conclusion

In this chapter, we built on everything learned in @sec-pipelines and @sec-pipelines-nonseq to look at concrete usage of pipelines for data preprocessing.
We focused primarily on feature engineering, which can make use of `r mlr3pipelines` to automate preprocessing as much as possible while still ensuring user control.
We looked at factor encoding for categorical variables, imputing missing data, transforming variables, and feature extraction.
Preprocessing is almost always required in machine learning experiments, and applying the `ppl("robustify")` will help in many cases to simplify this process by applying the most common preprocessing steps, we will see this in use in @sec-large-benchmarking.

We have not introduced any new classes in this chapter, so instead @tbl-prepro-api lists the `r ref("PipeOp")`s and `r ref("Graph")`s we discussed.

| PipeOp/Graph | Description |
| -- | --- |
| `r ref("PipeOpRemoveConstants")` | Remove variables consisting of one value |
| `r ref("PipeOpCollapseFactors")` | Combine rare factor levels |
| `r ref("PipeOpEncodeImpact")` | Impact encoding |
| `r ref("PipeOpEncode")` | Other factor encoding methods |
| `r ref("PipeOpMissInd")` | Add an indicator column to track missing data |
| `r ref("PipeOpImputeHist")` | Impute missing data by sampling from a histogram |
| `r ref("PipeOpImputeOOR")` | Impute missing data with out-of-range values |
| `r ref("pipeline_robustify")` | Graph with common imputation and encoding methods |
| `r ref("pipeline_targettrafo")` | Graph to transform target during training and invert transformation during prediction |

: `PipeOp`s and `Graph`s discussed in this chapter. {#tbl-prepro-api}

## Exercises

We will consider a prediction problem similar to the one from this chapter, but using the King County Housing regression data instead (available with `tsk("kc_housing")`).
To evaluate the models, we again use 10-fold CV, mean absolute error and `lrn("regr.glmnet")`.
For now we will ignore the `date` column and simply remove it:

```{r preprocessing-027}
library("mlr3data")
kc_housing = tsk("kc_housing")
kc_housing$select(setdiff(kc_housing$feature_names, "date"))
```

1. Have a look at the features, are there any features which might be problematic? If so, change or remove them.
  Check the dataset and learner properties to understand which preprocessing steps you need to do.
2. Build a suitable pipeline that allows `glmnet` to be trained on the dataset.
  Construct a new `glmnet` model with `ppl("robustify")`.
  Compare the two pipelines in a benchmark experiment.
3. Now consider the `date` feature:
  How can you extract information from this feature in a way that `glmnet` can use?
  Does this improve the performance of your pipeline?
  Finally, consider the spatial nature of the dataset.
  Can you extract an additional feature from the lat / long coordinates?
  (Hint: Downtown Seattle has lat/long coordinates `47.605`/`122.334`).

::: {.content-visible when-format="html"}
`r citeas(chapter)`
:::