large-scale_benchmarking.qmd

---
aliases:
  - "/large-scale_benchmarking.html"
---

# Large-Scale Benchmarking {#sec-large-benchmarking}

{{< include ../../common/_setup.qmd >}}

`r chapter = "Large-Scale Benchmarking"`
`r authors(chapter)`

```{r large-scale_benchmarking-001_benchmarking-001}
#| include: false
#| cache: false
lgr::get_logger("mlr3oml")$set_threshold("off")
library(mlr3batchmark)
library(batchtools)
library(mlr3oml)
manual_path = here::here("book", "openml", "manual")
if (!dir.exists(manual_path)) {
  dir.create(manual_path, recursive = TRUE)
}
options(mlr3oml.cache = here::here("book", "openml", "cache"))
```

In machine learning, it is often difficult to evaluate methods using mathematical analysis alone.
Even when formal analyses can be successfully applied, it is often an open question whether real-world datasets satisfy the necessary assumptions for the theorems to hold.
Empirical `r index('benchmark experiments')` evaluate the performance of different algorithms on a wide range of datasets.
These empirical investigations are essential for understanding the capabilities and limitations of existing methods and for developing new and improved approaches.
Trustworthy benchmark experiments are often 'large-scale', which means they may make use of many datasets, measures, and learners.
Moreover, datasets must span a wide range of domains and problem types as conclusions can only be drawn about the kind of datasets on which the `r index('benchmark study', see = "benchmark experiment")` was conducted.

Large-scale benchmark experiments consist of three primary steps: sourcing the data for the experiment, executing the experiment, and analyzing the results; we will discuss each of these in turn.
In @sec-openml we will begin by discussing `r ref_pkg("mlr3oml")`, which provides an interface between `r mlr3` and OpenML [@openml2013], a popular tool for uploading and downloading datasets.
Increasing the number of datasets leads to 'large-scale' experiments that may require significant computational resources, so in @sec-hpc-exec we will introduce `r ref_pkg("mlr3batchmark")`, which connects `mlr3` with `r ref_pkg("batchtools")` [@batchtools], which provides methods for managing and executing experiments on `r index('high-performance computing')` (HPC) clusters.
Finally, in @sec-benchmark-analysis we will demonstrate how to make use of `r ref_pkg("mlr3benchmark")` to formally analyze the results from large-scale benchmark experiments.

Throughout this chapter, we will use the running example of benchmarking a random forest model against a logistic regression as in @couronne2018random.
We will also assume that you have read @sec-pipelines and @sec-technical.
We make use of `ppl("robustify")` (@sec-prepro-robustify) for automating common preprocessing steps.
We also set a featureless baseline as a fallback learner (@sec-fallback) and set `"try"` as our encapsulation method (@sec-encapsulation), which logs errors/warnings to an external file that can be read by `batchtools` (we will return to this in @sec-batchtools-monitoring).

```{r large-scale_benchmarking-002_benchmarking-002, warning = FALSE}
# featureless baseline
lrn_baseline = lrn("classif.featureless", id = "featureless")

# logistic regression pipeline
lrn_lr = lrn("classif.log_reg")
lrn_lr = as_learner(ppl("robustify", learner = lrn_lr) %>>% lrn_lr)
lrn_lr$id = "logreg"
lrn_lr$encapsulate("try", fallback = lrn_baseline)

# random forest pipeline
lrn_rf = lrn("classif.ranger")
lrn_rf = as_learner(ppl("robustify", learner = lrn_rf) %>>% lrn_rf)
lrn_rf$id = "ranger"
lrn_rf$encapsulate("try", fallback = lrn_baseline)

learners = list(lrn_lr, lrn_rf, lrn_baseline)
```

As a starting example, we will compare our learners across three classification tasks using accuracy and three-fold CV.

```{r large-scale_benchmarking-003}
#| warning: false
design = benchmark_grid(tsks(c("german_credit", "sonar", "diabetes")),
  learners, rsmp("cv", folds = 10))
bmr = benchmark(design)
bmr$aggregate(msr("classif.acc"))[, .(task_id, learner_id, classif.acc)]
```

In this small experiment, random forests appears to outperform the other learners on all three datasets.
However, this analysis is not conclusive as we only considered three tasks, and the performance differences might not be statistically significant.
In the following, we will introduce some techniques to improve the study.

## Getting Data with OpenML {#sec-openml}

To draw meaningful conclusions from benchmark experiments, a good choice of datasets and tasks is essential.
`r index('OpenML', aside = TRUE, lower = FALSE)` is an open-source platform that facilitates the sharing and dissemination of machine learning research data, algorithms, and experimental results, in a standardized format enabling consistent cross-study comparison.
OpenML's design ensures that all data on the platform is '`r index('FAIR', lower = FALSE)`' (**F**indability, **A**ccessibility, **I**nteroperability and **R**eusability), which ensures the data is easily discoverable and reusable.
All entities on the platform have unique identifiers and standardized (meta)data that can be accessed via a REST API or the web interface.

In this section, we will cover some of the main features of OpenML and how to use them via the  `r mlr3oml` interface package.
In particular, we will discuss OpenML datasets, tasks, and task collections, but will not cover algorithms or experiment results here.

### Datasets {#sec-openml-dataset}

Finding data from OpenML is possible via the website or its REST API that `mlr3oml` interfaces.
`r ref("list_oml_data()")` can be used to filter datasets for specific properties, for example by number of features, rows, or number of classes in a classification problem:

```{r large-scale_benchmarking-004}
#| include: false
path_odatasets = here::here("book", "openml", "manual", "odatasets_filter.rds")
```

```{r large-scale_benchmarking-005_benchmarking-014, eval = !file.exists(path_odatasets)}
library(mlr3oml)

odatasets = list_oml_data(
  number_features = c(10, 20),
  number_instances = c(45000, 50000),
  number_classes = 2
)
```

```{r large-scale_benchmarking-006}
#| include: false
if (file.exists(path_odatasets)) {
  odatasets = readRDS(path_odatasets)
} else {
  saveRDS(odatasets, path_odatasets)
}
```


```{r large-scale_benchmarking-007}
odatasets[NumberOfFeatures < 16,
  c("data_id", "name", "NumberOfFeatures", "NumberOfInstances")]
```

Note that `list_oml_data()` returns a `data.table` with many more meta-features than shown here; this table can itself be used to filter further.

We can see that some datasets have duplicated names, which is why each dataset also has a unique ID.
By example, let us consider the 'adult' dataset with ID 1590.
Metadata for the dataset is loaded with `r ref("odt()", aside = TRUE)`, which returns an object of class `r ref("OMLData")`.

```{r large-scale_benchmarking-008_benchmarking-003}
odata = odt(id = 1590)
odata
```

The `OMLData` object contains metadata about the dataset but importantly does not (yet) contain the data.
This means that information about the dataset can be queried without having to load the entire data into memory, for example, the license and dimension of the data:

```{r large-scale_benchmarking-009_benchmarking-004}
odata$license
c(nrow = odata$nrow, ncol = odata$ncol)
```

If we want to work with the actual data, then accessing the `$data` field will download the data, import it into R, and then store the `data.frame` in the `OMLData` object:

```{r large-scale_benchmarking-010_benchmarking-005}
# first 5 rows and columns
odata$data[1:5, 1:5]
```

:::{.callout-tip}
## mlr3oml Cache

After `$data` has been called the first time, all subsequent calls to `$data` will be transparently redirected to the in-memory `data.frame`.
Additionally, many objects can be permanently cached on the local file system by setting the option `mlr3oml.cache` to either `TRUE` or to a specific path to be used as the cache folder.
:::

Data can then be converted into `mlr3` backends (see @sec-backends) with the `r ref("as_data_backend()")` function and then into tasks:

```{r large-scale_benchmarking-011_benchmarking-006}
backend = as_data_backend(odata)
tsk_adult = as_task_classif(backend, target = "class")
tsk_adult
```

Some datasets on OpenML contain columns that should neither be used as a feature nor a target.
The column names that are usually included as features are accessible through the field `$feature_names`, and we assign them to the `mlr3` task accordingly.
Note that for the dataset at hand, this would not have been necessary, as all non-target columns are to be treated as predictors, but we include it for clarity.

```{r large-scale_benchmarking-012}
tsk_adult$col_roles$feature = odata$feature_names
tsk_adult
```

### Task {#sec-openml-task}

OpenML tasks are built on top of OpenML datasets and additionally specify the target variable, the train-test splits to use for resampling, and more.
Note that this differs from `mlr3` `Task` objects, which do not contain information about the resampling procedure.
Similarly to `mlr3`, OpenML has different types of tasks, such as regression and classification.
Analogously to filtering datasets, tasks can be filtered with `r ref("list_oml_tasks()")`.
To find a task that makes use of the data we have been using, we would pass the data ID to the `data_id` argument:

```{r large-scale_benchmarking-013}
#| include: false
path_adult_tasks = here::here("book", "openml", "manual", "adult_tasks.rds")
```

```{r large-scale_benchmarking-014, eval = !file.exists(path_adult_tasks)}
# tasks making use of the adult data
adult_tasks = list_oml_tasks(data_id = 1590)
```

```{r large-scale_benchmarking-015}
#| include: false
if (file.exists(path_adult_tasks)) {
  adult_tasks = readRDS(path_adult_tasks)
} else {
  saveRDS(adult_tasks, path_adult_tasks)
}
```

```{r large-scale_benchmarking-016}
adult_tasks[task_type == "Supervised Classification", task_id]
```

From these tasks, we randomly select the task with ID 359983.
We can load the object using `r ref("otsk()", aside = TRUE)`, which returns an `r ref("OMLTask")` object.

```{r large-scale_benchmarking-017_benchmarking-009}
otask = otsk(id = 359983)
otask
```

The `OMLData` object associated with the underlying dataset can be accessed through the `$data` field.

```{r large-scale_benchmarking-018_benchmarking-010}
otask$data
```

The data splits associated with the estimation procedure are accessible through the field `$task_splits`.
In `mlr3` terms, these are the instantiation of a `r ref("mlr3::Resampling")` on a specific `r ref("mlr3::Task")`.

```{r large-scale_benchmarking-019_benchmarking-011}
otask$task_splits
```

The OpenML task can be converted to both an `mlr3::Task` and `r ref("mlr3::ResamplingCustom")` instantiated on the task using `r ref("as_task()")` and `r ref("as_resampling()")`, respectively:

```{r large-scale_benchmarking-020_benchmarking-012}
tsk_adult = as_task(otask)
tsk_adult

resampling = as_resampling(otask)
resampling
```

`mlr3oml` also allows direct construction of `mlr3` tasks and resamplings with the standard `r ref("tsk()")` and `r ref("rsmp()")` constructors, e.g.:

```{r large-scale_benchmarking-021}
tsk("oml", task_id = 359983)
```


### Task Collection {#sec-openml-collection}

The OpenML task collection is a container object bundling existing tasks.
This allows for the creation of `r index("benchmark suites")`, which are curated collections of tasks that satisfy certain quality criteria.
Examples include the OpenML CC-18 benchmark suite [@bischl2021openml], the AutoML benchmark [@amlb2022] and the benchmark for tabular deep learning [@grinsztajn2022why].
`r ref("OMLCollection")` objects are loaded with `r ref("ocl()", aside = TRUE)`, by example we will look at CC-18, which has ID 99:

```{r large-scale_benchmarking-022}
#| include: false
path_otask_collection = here::here("book", "openml", "manual", "otask_collection99.rds")
```

```{r large-scale_benchmarking-023_benchmarking-017, eval = !file.exists(path_otask_collection)}
otask_collection = ocl(id = 99)
```

```{r large-scale_benchmarking-024}
#| include: false
if (file.exists(path_otask_collection)) {
  otask_collection = readRDS(path_otask_collection)
} else {
  # need to trigger the download
  otask_collection$task_ids
  saveRDS(otask_collection, path_otask_collection)
}
```

```{r large-scale_benchmarking-025_benchmarking-019}
otask_collection
```

The task includes 72 classification tasks on different datasets that can be accessed through `$task_ids`:

```{r large-scale_benchmarking-026_benchmarking-020}
otask_collection$task_ids[1:5] # first 5 tasks in the collection
```

Task collections can be used to quickly define benchmark experiments in `mlr3`.
To easily construct all tasks and resamplings from the benchmarking suite, you can use `r ref("as_tasks()", index = TRUE)` and `r ref("as_resamplings()", index = TRUE)` respectively:

```{r large-scale_benchmarking-027, eval = FALSE}
tasks = as_tasks(otask_collection)
resamplings = as_resamplings(otask_collection)
```

Alternatively, if we wanted to filter the collection further, say to a binary classification experiment with six tasks, we could run `r ref("list_oml_tasks()")` with the task IDs from the CC-18 collection as argument `task_id`.
We can either use the `list_oml_tasks()` argument to request the number of classes to be `2`, or we can make use of the fact that the result of `list_oml_tasks()` is a `data.table` and subset the resulting table.

```{r large-scale_benchmarking-028}
#| include: false
path_binary_cc18 = here::here("book", "openml", "manual", "binary_cc18.rds")
```

```{r large-scale_benchmarking-029_benchmarking-021, eval = !file.exists(path_binary_cc18)}
binary_cc18 = list_oml_tasks(
  limit = 6,
  task_id = otask_collection$task_ids,
  number_classes = 2
)
```

```{r large-scale_benchmarking-030}
#| include: false
if (!file.exists(path_binary_cc18)) {
  saveRDS(binary_cc18, path_binary_cc18)
} else {
  binary_cc18 = readRDS(path_binary_cc18)
}
```

We now define the tasks and resamplings which we will use for comparing the logistic regression with the random forest learner.
Note that all resamplings in this collection consist of exactly 10 iterations.

```{r large-scale_benchmarking-031_benchmarking-024}
# load tasks as a list
otasks = lapply(binary_cc18$task_id, otsk)

# convert to mlr3 tasks and resamplings
tasks = as_tasks(otasks)
resamplings = as_resamplings(otasks)
```

To define the design table, we use `r ref("benchmark_grid()")` and set `paired` to `TRUE`, which is used in situations where each resampling is instantiated on a corresponding task (therefore the `tasks` and `resamplings` below must have the same length) and each learner should be evaluated on every resampled task.

```{r large-scale_benchmarking-032_benchmarking-025}
large_design = benchmark_grid(tasks, learners, resamplings,
  paired = TRUE)
large_design[1:6] # first 6 rows
```

Having set up our large experiment, we can now look at how to efficiently carry it out on a cluster.

## Benchmarking on HPC Clusters {#sec-hpc-exec}

As discussed in @sec-parallelization, parallelization of benchmark experiments is straightforward as they are `r index('embarrassingly parallel')`.
However, for large experiments, parallelization on a `r index('high-performance computing', aside = TRUE)` (HPC) cluster is often preferable.
`r ref_pkg("batchtools")` provides a framework to simplify running large batches of computational experiments in parallel from R on such sites.
It is highly flexible, making it suitable for a wide range of computational experiments, including machine learning, optimization, simulation, and more.

:::{.callout-tip}
## `"batchtools"` backend for `future`

In @sec-parallel-resample we touched upon different parallelization backends.
The package `r ref_pkg("future")` includes a `"batchtools"` plan, however, this does not allow the additional control that comes with working with `batchtools` directly.
:::

An HPC cluster is a collection of interconnected computers or servers providing computational power beyond what a single computer can achieve.
HPC clusters typically consist of multiple compute nodes, each with multiple CPU/GPU cores, memory, and local storage.
These nodes are usually connected by a high-speed network and network file system which enables the nodes to communicate and work together on a given task.
The most important difference between HPC clusters and a personal computer (PC), is that the nodes often cannot be accessed directly, but instead, computational jobs are queued by a `r index("scheduling system")` such as Slurm (Simple Linux Utility for Resource Management).
A scheduling system is a software tool that orchestrates the allocation of computing resources to users or applications on the cluster.
It ensures that multiple users and applications can access the resources of the cluster fairly and efficiently, and also helps to maximize the utilization of the computing resources.

@fig-hpc contains a rough sketch of an HPC architecture.
Multiple users can log into the head node (typically via SSH) and add their `r index('computational job')`s to the queue by sending a command of the form "execute computation X using resources Y for Z amount of time".
The scheduling system controls when these computational jobs are executed.

For the rest of this section, we will look at how to use `batchtools` and `r mlr3batchmark` for submitting jobs, adapting jobs to clusters, ensuring reproducibility, querying job status, and debugging failures.

```{r large-scale_benchmarking-033_benchmarking-026, echo = FALSE}
#| label: fig-hpc
#| fig-cap: "Illustration of an HPC cluster architecture."
#| fig-alt: "Flow diagram of objects. Left is a laptop with an arrow to an object that says 'Head Node - Scheduler', the arrow has text 'SSH'. The scheduler has a bidirectional arrow with text 'Submit' to 'Queue' that has an arrow to 'Computing Nodes'. The scheduler also has an arrow to 'File System' which has a double arrow connecting it to/from the 'Computing Nodes' object with text 'Data'."
include_multi_graphics("mlr3book_figures-32")
```

### Experiment Registry Setup {#sec-registry}

`batchtools`\index{\texttt{batchtools}} is built around experiments or '`r index("jobs")`'.
One replication of a job is defined by applying a (parameterized) algorithm to a (parameterized) problem.
A benchmark experiment in `batchtools` consists of running many such experiments with different algorithms, algorithm parameters, problems, and problem parameters.
Each such experiment is computationally independent of all other experiments and constitutes the basic level of computation `batchtools` can parallelize.
For this section, we will define a single `batchtools` experiment as one resampling iteration of one learner on one task, in @sec-custom-experiments we will look at different ways of defining an experiment.

The first step in running an experiment is to create or load an experiment registry with `r ref("makeExperimentRegistry()")` or `r ref("loadRegistry()")` respectively.
This constructs the inter-communication object for all functions in `batchtools` and corresponds to a folder on the file system.
Among other things, the experiment registry stores the algorithms, problems, and job definitions; log outputs and status of submitted, running, and finished jobs; job results; and the "cluster function" that defines the interaction with the scheduling system in a scheduling-software-agnostic way.

Below, we create a registry in a subdirectory of our working directory -- on a real cluster, make sure that this folder is stored on a shared network filesystem, otherwise, the nodes cannot access it.
We also set the registry's `seed` to `1` and the `packages` to `"mlr3verse"`, which will make these packages available in all our experiments.

```{r large-scale_benchmarking-034, echo = FALSE}
#| cache: false
if (dir.exists("experiments")) {
  unlink("experiments", recursive = TRUE)
}
```

```{r large-scale_benchmarking-035_benchmarking-027, message=FALSE, warning=FALSE}
#| cache: false
library(batchtools)

# create registry
reg = makeExperimentRegistry(
  file.dir = "./experiments",
  seed = 1,
  packages = "mlr3verse"
)
```

Once the registry has been created, we need to populate it with problems and algorithms to form the jobs, this is most easily carried out with `mlr3batchmark`\index{\texttt{mlr3batchmark}}, although finer control is possible with `batchtools` and will be explored in @sec-custom-experiments.
`r ref("batchmark()", aside = TRUE)` converts `mlr3` tasks and resamplings to `batchtools` problems, and converts `mlr3` learners to `batchtools` algorithms; jobs are then created for all resampling iterations.

```{r large-scale_benchmarking-036_benchmarking-029}
#| cache: false
#| output: false
library(mlr3batchmark)
batchmark(large_design, reg = reg)
```

Now the registry includes six problems, one for each resampled task, and $180$ jobs from $3$ learners $\times$ $6$ tasks $\times$ $10$ resampling iterations.
The single algorithm in the registry is because `mlr3batchmark` specifies a single algorithm that is parametrized with the learner IDs.


```{r large-scale_benchmarking-037_benchmarking-030}
reg
```

By default, the "Interactive" cluster function (see `r ref("makeClusterFunctionsInteractive()")`) is used -- this is the abstraction for the scheduling system, and "interactive" here means to not use a real scheduler but instead to use the interactive R session for sequential computation.
`r ref("getJobTable()")` can be used to get more detailed information about the jobs.
Here, we only show a few selected columns for readability and unpack the list columns `algo.pars` and `prob.pars` using `r ref("unwrap()")`.

```{r large-scale_benchmarking-038_benchmarking-032}
job_table = getJobTable(reg = reg)
job_table = unwrap(job_table)
job_table = job_table[,
  .(job.id, learner_id, task_id, resampling_id, repl)
]

job_table
```

In this output, we can see how each job is now assigned a unique `job.id` and that each row corresponds to a single iteration (column `repl`) of a resample experiment.

### Job Submission {#sec-batchtools-submission}

With the experiments defined, we can now submit them to the cluster.
However, it is best practice to first test each algorithm individually using `r ref("testJob()", aside = TRUE)`.
By example, we will only test the first job (`id = 1`) and will use an external R session (`external = TRUE`).

```{r large-scale_benchmarking-039_benchmarking-033}
#| eval: false
result = testJob(1, external = TRUE, reg = reg)
```

Once we are confident that the jobs are defined correctly (see @sec-batchtools-monitoring for jobs with errors), we can proceed with their submission, by specifying the resource requirements for each computational job and then optionally grouping jobs.

Configuration of resources is dependent on the cluster function set in the registry.
We will assume we are working with a `r index('Slurm', lower = FALSE)` cluster and accordingly initialize the cluster function with `r ref("makeClusterFunctionsSlurm()")` and will make use of the `slurm-simple.tml` template file that can be found in a subdirectory of the `batchtools` package itself (the exact location can be found by running `system.file("templates", package = "batchtools")`), or the `batchtools` GitHub repository.
A template file is a shell script with placeholders filled in by `batchtools` and contains the command to start the computation via `Rscript` or `R CMD batch`, as well as comments which serve as annotations for the scheduler, for example, to communicate resources or paths on the file system.

The exemplary template should work on many Slurm installations out-of-the-box, but you might have to modify it for your cluster -- it can be customized to work with more advanced configurations.

```{r large-scale_benchmarking-040_benchmarking-034}
cf = makeClusterFunctionsSlurm(template = "slurm-simple")
```
```{r large-scale_benchmarking-041}
#| include: false
#| cache: false
cf = makeClusterFunctionsInteractive()
```

To proceed with the examples on a local machine, we recommend setting the cluster function to a Socket backend with `r ref("makeClusterFunctionsSocket()")`.
The chosen cluster function can be saved to the registry by passing it to the `$cluster.functions` field.

```{r large-scale_benchmarking-042_benchmarking-036, output = FALSE}
#| cache: false
reg$cluster.functions = cf
saveRegistry(reg = reg)
```

With the registry setup, we can now decide if we want to run the experiments in chunks (@sec-parallelization) and then specify the resource requirements for the submitted jobs.

For this example, we will use `r ref("chunk()", aside = TRUE)` to `r index('chunk')` the jobs such that five iterations of one resample experiment are run sequentially in one computational job -- in practice the optimal grouping will be highly dependent on your experiment (@sec-parallelization).

```{r large-scale_benchmarking-043_benchmarking-037}
ids = job_table$job.id
chunks = data.table(
  job.id = ids, chunk = chunk(ids, chunk.size = 5, shuffle = FALSE)
)
chunks[1:6] # first 6 jobs
```

The final step is to decide the resource requirements for each job.
The set of resources depends on your cluster and the corresponding template file.
If you are unsure about the resource requirements, you can start a subset of jobs with liberal resource constraints, e.g. the maximum runtime allowed for your computing site.
Measured runtimes and memory usage can later be queried with `r ref("getJobTable()")` and used to better estimate the required resources for the remaining jobs.
In this example we will set the number of CPUs per job to `1`, the walltime (time limit before jobs are stopped by the scheduler) to one hour (`3600` seconds), and the RAM limit (memory limit before jobs are stopped by the scheduler) to `8000` megabytes.

```{r large-scale_benchmarking-044}
resources = list(ncpus = 1, walltime = 3600, memory = 8000)
```

With all the elements in place, we can now submit our jobs.

<!-- output to false otherwise we see the interactive cluster function -->
```{r large-scale_benchmarking-045_benchmarking-038, output = FALSE}
#| cache: false
submitJobs(ids = chunks, resources = resources, reg = reg)

# wait for all jobs to terminate
waitForJobs(reg = reg)
```

:::{.callout-tip}
## Submitting Jobs
A good approach to submit computational jobs is by using a persistent R session (e.g., with Terminal Multiplexer (TMUX)) on the head node to continue job submission (or computation, depending on the cluster functions) in the background.

However, `batchtools` registries are saved to the file system and therefore persistent when the R session is terminated.
This means that you can also submit jobs from an interactive R session, terminate the session, and analyze the results later in a new session.
:::

###  Job Monitoring, Error Handling, and Result Collection {#sec-batchtools-monitoring}

Once jobs have been submitted, they can then be queried with `r ref("getStatus()", aside = TRUE)` to find their current status and the results (or errors) can be investigated.
If you terminated your R sessions after job submission, you can load the experiment registry with `r ref("loadRegistry()", aside = TRUE)`.

```{r large-scale_benchmarking-046_benchmarking-041}
getStatus(reg = reg)
```

To query the ids of jobs in the respective categories, see `r ref("findJobs()")` and, e.g., `r ref("findNotSubmitted()")` or `r ref("findDone()")`.
In our case, we can see all experiments finished and none expired (i.e., were removed from the queue without ever starting, `Expired : 0`) or crashed (`Error : 0`).
It can still be sensible to use `r ref("grepLogs()")` to check the logs for suspicious messages and warnings before proceeding with the analysis of the results.

In any large-scale experiment many things can and will go wrong, for example, the cluster might have an outage, jobs may run into resource limits or crash, or there could be bugs in your code.
In these situations, it is important to quickly determine what went wrong and to recompute only the minimal number of required jobs.

To see `r index('debugging')` in practice we will use the debug learner (see @sec-error-handling) with a 50% probability of erroring in training.
When calling `r ref("batchmark()")` again, the new experiments will be added to the registry on top of the existing jobs.

```{r large-scale_benchmarking-047, output = FALSE}
#| cache: false
extra_design = benchmark_grid(tasks,
  lrn("classif.debug", error_train = 0.5), resamplings, paired = TRUE)

batchmark(extra_design, reg = reg)
```

::: {.callout-tip}
## Registry Argument
All `batchtools` functions that interoperate with a registry take a registry as an argument.
By default, this argument is set to the last created registry, which is currently the `reg` object defined earlier.
We pass it explicitly in this section for clarity.
:::

Now we can get the IDs of the new jobs (which have not been submitted yet) and submit them by passing their IDs.

```{r large-scale_benchmarking-048, output = FALSE}
#| cache: false
ids = findNotSubmitted(reg = reg)
submitJobs(ids, reg = reg)
```

```{r large-scale_benchmarking-049, include=FALSE}
waitForJobs(reg = reg)
```

After these jobs have terminated, we can get a summary of those that failed:

```{r large-scale_benchmarking-050}
getStatus(reg = reg)

error_ids = findErrors(reg = reg)
summarizeExperiments(error_ids, by = c("task_id", "learner_id"),
  reg = reg)
```

In a real experiment, we would now investigate the debug learner further to understand why it errored, try to fix those bugs, and then potentially rerun those experiments only.

Assuming learners have been debugged (or we are happy to ignore them), we can then collect the results of our experiment with `r ref("mlr3batchmark::reduceResultsBatchmark()")`, which constructs a `r ref("BenchmarkResult")` from the results.
Below we filter out results from the debug learner.

```{r large-scale_benchmarking-051}
ids = findExperiments(algo.pars = learner_id != "classif.debug",
  reg = reg)
bmr = reduceResultsBatchmark(ids, reg = reg)
bmr$aggregate()[1:5]
```

### Custom Experiments with batchtools {#sec-custom-experiments}

{{< include ../../common/_optional.qmd >}}

In general, we recommend using `mlr3batchmark` for scheduling simpler `mlr3` jobs on an HPC, however, we will also briefly show you how to use `batchtools` without `mlr3batchmark` for finer control over your experiment.
Again we start by creating an experiment registry.

```{r large-scale_benchmarking-052, include = FALSE}
#| cache: false
if (dir.exists("experiments-custom")) {
  unlink("experiments-custom", recursive = TRUE)
}
```

```{r large-scale_benchmarking-053_benchmarking-046, message = FALSE}
#| cache: false
reg = makeExperimentRegistry(
  file.dir = "./experiments-custom",
  seed = 1,
  packages = "mlr3verse"
)
```

"Problems" are then manually registered with `r ref("addProblem()")`.
In this example, we will register all task-resampling combinations of the `large_design` above using the task ids as unique names.
We specify that the `data` for the problem (i.e., the static data that is trained/tested by the learner) is the task/resampling pair.
Finally, we pass a function (`fun`, dynamic problem part) that takes in the static problem `data` and returns it as the problem `instance` without making changes (@fig-batchtools-illustration).
The `fun` shown below is the default behavior and could be omitted, we show it here for clarity.
This function could be more complex and take further parameters to modify the problem instance dynamically.

```{r large-scale_benchmarking-054_benchmarking-047, output = FALSE}
#| cache: false
for (i in seq_along(tasks)) {
  addProblem(
    name = tasks[[i]]$id,
    data = list(task = tasks[[i]], resampling = resamplings[[i]]),
    fun = function(data, job, ...) data,
    reg = reg
  )
}

```

```{r large-scale_benchmarking-055_benchmarking-051, echo = FALSE}
#| label: fig-batchtools-illustration
#| fig-cap: "Illustration of a batchtools problem, algorithm, and experiment. "
#| fig-alt: "The diagram shows a rectangle that says 'static problem part, data', with an arrow pointing to 'dynamic problem function, fun(data, ...)' and 'algorithm function, fun(data, instance, ...)'. A box that says 'problem design, (addExperiments)' also has an arrow to the 'dynamic...' box. The 'dynamic...' box then has an arrow with text 'instance' that points to the 'algorithm function' box. A box that says 'algorithm design, (addExperiments)' also points to the 'algorithm function' box. Finally the 'algorithm function' box points to 'result'."
include_multi_graphics("mlr3book_figures-31")
```

Next, we need to specify the algorithm to run with `r ref("addAlgorithm()")`.
Algorithms are again specified with a unique `name`, as well as a function to define the computational steps of the experiment and to return its result.

Here, we define one job to represent a complete resample experiment.
In general, algorithms in `batchtools` may return arbitrary objects -- those are simply stored on the file system and can be processed with a custom function while collecting the results.

```{r large-scale_benchmarking-056_benchmarking-048, message=FALSE}
#| cache: false
addAlgorithm(
  "run_learner",
  fun = function(instance, learner, job, ...) {
    resample(instance$task, learner, instance$resampling)
  },
  reg = reg
)
```

Finally, we will define concrete experiments with `r ref("addExperiments()")` by passing problem designs (`prob.designs`) and algorithm designs (`algo.designs`) that assign parameters to problems and algorithms, respectively (@fig-batchtools-illustration).

In the code below, we add all resampling iterations for the six tasks as experiments.
By leaving `prob.designs` unspecified, experiments for all existing problems are created per default.
We set the `learner` parameter of our algorithm (`"run_learner"`) to be the three learners from our `large_design` object.
Note that whenever an experiment is added, the current seed is assigned to the experiment and then incremented.

```{r large-scale_benchmarking-057_benchmarking-049, eval = FALSE}
alg_des = list(run_learner = data.table(learner = learners))
addExperiments(algo.designs = alg_des, reg = reg)
summarizeExperiments()
```
```{r large-scale_benchmarking-058, include = FALSE, output = FALSE}
#| cache: false
alg_des = list(run_learner = data.table(learner = learners))
addExperiments(algo.designs = alg_des, reg = reg)
```
```{r large-scale_benchmarking-059, include = FALSE, output = TRUE}
#| cache: false
summarizeExperiments()
```

Our jobs can now be submitted to the cluster; by not specifying specific job IDs, *all* experiments are submitted.

```{r large-scale_benchmarking-060, output = FALSE}
#| cache: false
submitJobs(reg = reg)
```

```{r large-scale_benchmarking-061, include = FALSE}
#| cache: false
waitForJobs(reg = reg)
```

We can retrieve the job results using `r ref("loadResult()")`, which outputs the objects returned by the algorithm function, which in our case is a `r ref("ResampleResult")`.
To retrieve all results at once, we can use `r ref("reduceResults()")` to create a single `r ref("BenchmarkResult")`.
For this, we use the combine function `c()` which can combine multiple objects of type `ResampleResult` or `BenchmarkResult` to a single `BenchmarkResult`.

```{r large-scale_benchmarking-062_benchmarking-054}
rr = loadResult(1, reg = reg)
as.data.table(rr)[1:5]

bmr = reduceResults(c, reg = reg)
bmr$aggregate()[1:5]
```

## Statistical Analysis {#sec-benchmark-analysis}

The final step of a benchmarking experiment is to use statistical tests to determine which (if any) of our learners performed the best.
`r ref("mlr3benchmark")` provides infrastructure for applying statistical significance tests on `r ref("BenchmarkResult")` objects.

Currently, `r index('Friedman')` tests and pairwise Friedman-Nemenyi tests [@demsar2006] are supported to analyze benchmark experiments with at least two independent tasks and at least two learners.
As a first step, we recommend performing a pairwise comparison of learners using pairwise Friedman-Nemenyi tests with `$friedman_posthoc()`.
This method first performs a global comparison to see if any learner is statistically better than another.
To use these methods we first convert the benchmark result to a `r ref("BenchmarkAggr")` object using `r ref("as_benchmark_aggr()", aside = TRUE)`.

```{r large-scale_benchmarking-063_benchmarking-056}
library(mlr3benchmark)
bma = as_benchmark_aggr(bmr, measures = msr("classif.ce"))
bma$friedman_posthoc()
```

These results indicate a statistically significant difference between the `"featureless"` learner and `"ranger"` (assuming $p\leq0.05$ is significant).
This table can be visualized in a critical difference plot (@fig-lsb-cd), which typically shows the mean rank of a learning algorithm on the x-axis along with a thick horizontal line that connects learners that are pairwise not significantly different (while correcting for multiple tests).

```{r large-scale_benchmarking-064_benchmarking-057, fig.height = 1.5}
#| label: fig-lsb-cd
#| fig-cap: "Critical difference diagram comparing the random forest, logistic regression, and featureless baseline. The critical difference of 1.35 in the title refers to the difference in mean rank required to conclude that one learner performs statistically different to another."
#| fig-alt: "Figure shows a one-axis diagram ranging from 0 to 4, above the diagram is a thick black line with text 'Critical Difference = 1.35'. Diagram shows 'ranger' on the far left just to the right of '1', then 'logreg' just to the left of '2', then 'featureless' just under '3'. There is a thick, black line connecting 'ranger' and 'logreg', as well as a thick, black line connecting 'logreg' and 'featureless'."
autoplot(bma, type = "cd", ratio = 1/5)
```

Using @fig-lsb-cd we can conclude that on average the random forest had the lowest (i.e., best) rank, followed by the logistic regression, and then the featureless baseline.
While the random forest was statistically better performing than the baseline (no connecting line in @fig-lsb-cd), it was not statistically superior to the logistic regression (connecting line in @fig-lsb-cd).
We could now further compare this with the large benchmark study conducted by @couronne2018random, where the random forest outperformed the logistic regression in 69% of 243 real-world datasets.

## Conclusion

In this chapter, we have explored how to conduct large-scale machine learning experiments using `mlr3`.
We have shown how to acquire diverse datasets from OpenML through the `r mlr3oml` interface package, how to execute large-scale experiments with `batchtools` and `r mlr3batchmark` integration, and finally how to analyze the results of these experiments with `r mlr3benchmark`.
For further reading about `r ref_pkg("batchtools")` we recommend @batchtools and @JSSv064i11.

| Class | Constructor/Function | Fields/Methods |
| --- | --- | --- |
| `r ref("OMLData")` | `r ref("odt()")` | `$data`; `$feature_names` |
| `r ref("OMLTask")` | `r ref("otsk()")` | `$data`; `$task_splits` |
| `r ref("OMLCollection")` | `r ref("ocl()")` | `$task_ids` |
| `Registry` | `r ref("makeExperimentRegistry()")` | `r ref("submitJobs()")`; `r ref("getStatus()")`; `r ref("reduceResultsBatchmark")`; `r ref("getJobTable")` |
| | `r ref("batchmark()")` | - |
| `r ref("BenchmarkAggr()")` | `r ref("as_benchmark_aggr()")` | `$friedman_posthoc()` |

: Important classes and functions covered in this chapter with underlying class (if applicable), class constructor or function, and important class fields and methods (if applicable). {#tbl-api-large-benchmarking}

## Exercises

In these exercises, we will conduct an empirical study analyzing whether a random forest is predictively stronger than a single decision tree.
  Our null hypothesis is that there is no significant performance difference.

1. Load the OpenML collection with ID 269, which contains regression tasks from the AutoML benchmark [@amlb2022].
  Peek into this suite to study the contained data sets and their characteristics.
  Then find all tasks with less than 4000 observations and convert them to `mlr3` tasks.
2. Create an experimental design that compares `lrn("regr.ranger")` and `lrn("regr.rpart")` on those tasks.
  Use the robustify pipeline for both learners and a featureless fallback learner.
  You can use three-fold CV instead of the OpenML resamplings to save time.
  Run the comparison experiments with `batchtools`.
  Use default hyperparameter settings and do not perform any tuning to keep the experiments simple.
3. Conduct a global Friedman test and, if appropriate, post hoc Friedman-Nemenyi tests, and interpret the results.
  As an evaluation measure, use the MSE.

::: {.content-visible when-format="html"}
`r citeas(chapter)`
:::