eval-performance.Rmd

---
title: "Evaluation of performances"
output: 
  workflowr::wflow_html:
    includes:
      in_header: header.html
editor_options:
  chunk_output_type: console
author: "Patrick Schratz"
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  fig.retina = 3,
  fig.align = "center",
  fig.width = 6.93,
  fig.height = 6.13,
  out.width = "100%",
  echo = FALSE
)

R.utils::sourceDirectory("R")

# load drake objects
drake::loadd(
  bm_aggregated
)
library("mlr")
library("magrittr")
library("dplyr")
library("ggplot2")
library("ggrepel")
library("ggsci")
library("ggpubr")
```

### Aggregated performances

#### (Table) All leaner/filter/task combinations ordered by performance.

```{r eval-performance-1}
getBMRAggrPerformances(bm_aggregated, as.df = TRUE) %>%
  mutate(task.id = recode_factor(task.id,
    `defoliation-all-plots-HR` = "HR",
    `defoliation-all-plots-VI` = "VI",
    `defoliation-all-plots-NRI` = "NRI"
  )) %>%
  tidyr::separate(learner.id, c("learner_group", "filter"),
    remove = FALSE,
    sep = " "
  ) %>%
  group_by(learner_group, task.id, filter) %>%
  summarise(best = min(rmse.test.rmse)) %>%
  mutate(filter = replace(filter, is.na(filter), "No Filter")) %>%
  arrange(best) %>%
  knitr::kable()
```

#### (Table) Best learner/filter/task combination

```{r eval-performance-2}
getBMRAggrPerformances(bm_aggregated, as.df = TRUE) %>%
  tidyr::separate(learner.id, c("learner_group", "filter"),
    remove = FALSE,
    sep = " "
  ) %>%
  group_by(learner_group) %>%
  slice(which.min(rmse.test.rmse)) %>%
  mutate(filter = replace(filter, is.na(filter), "No Filter")) %>%
  arrange(rmse.test.rmse) %>%
  select(-6) %>%
  knitr::kable()
```

#### (Plot) Best learner/filter combs for all tasks

```{r eval-performance-3}
results_aggr <- getBMRAggrPerformances(bm_aggregated, as.df = TRUE) %>%
  mutate(task.id = recode_factor(task.id,
    `defoliation-all-plots-HR` = "HR",
    `defoliation-all-plots-VI` = "VI",
    `defoliation-all-plots-NRI` = "NRI"
  )) %>%
  tidyr::separate(learner.id, c("learner_group"),
    remove = FALSE,
    sep = " "
  ) %>%
  group_by(learner_group, task.id) %>%
  summarise(best = min(rmse.test.rmse))

results_aggr %>%
  ggplot(aes(x = best, y = task.id)) +
  geom_point(aes(color = learner_group, size = 1.7)) +
  scale_color_nejm() +
  labs(x = "RMSE", y = "Task", color = "Learner") +
  guides(size = FALSE) +
  scale_x_continuous(limits = c(1.8, 4)) +
  geom_label_repel(
    label = round(results_aggr$best, 3),
    size = 3, nudge_y = 0.3
  ) +
  theme_pubr()
```