README.Rmd

---
output: 
  - dynbenchmark::pdf_supplementary_note
  - dynbenchmark::github_markdown_nested
---

```{r setup, include=FALSE}
library(tidyverse)
library(dynbenchmark)
```

# Metrics to compare two trajectories

A trajectory, as defined in our evaluation, is a model with multiple abstractions. The top abstraction is the topology which contains information about the paths each cell can take from their starting point. Deeper abstractions involve the mapping of each cell to a particular branch within this network, and the position (or ordering) of each cells within these branches. Internally, the topology is represented by the milestone network and regions of delayed commitment, the branch assignment and cellular positions are represented by the milestone percentages (`r ref("fig", "trajectory_model_example", prefix = "S")`).


```{r}
add_fig(
  raw_file("trajectory_model_example.svg", "02-metrics"),
  "trajectory_model_example",
  "An example trajectory that will be used throughout this section.",
  "It contains contains four milestones (W to Z) and five cells (a to e).",
  width = 12,
  height = 6
)
```

Given the multilayered complexity of a trajectory model, it is not trivial to compare the similarity of two trajectory models using only one metric. We therefore sought to use different comparison metrics, each serving a different purpose:

- **Specific metrics** investigate one particular aspect of the trajectory. Such metrics make it possible to find particular weak points for methods, e.g. that a method is very good at ordering but does not frequently find the correct topology. Moreover, having multiple individual metrics allow personalised rankings of methods, for example for users which are primarily interested in using the method correct topology.
- **Application metrics** focus on the quality of a downstream analysis using the trajectory. For example, it measures whether the trajectory can be used to find accurate differentially expressed genes.
- **Overall metrics** should capture all the different abstractions, in other words such metrics measure whether the resulting trajectory has a good topology, that the cells belong to similar branches _and_ that they are ordered correctly.

Here, we first describe and illustrate several possible specific, application and overall metrics. Next, we test these metrics on several test cases, to make sure they robustly identify "wrong" trajectory predictions.

All metrics described here were implemented within the [_dyneval_](https://github.com/dynverse/dyneval) R package ([https://github.com/dynverse/dyneval](https://github.com/dynverse/dyneval)).

```{r}
dynbenchmark::knit_nest("01-metric_characterisation/README.Rmd")
```

\clearpage

```{r}
dynbenchmark::knit_nest("02-metric_conformity/README.Rmd")
```

\clearpage

```{r}
dynbenchmark::knit_nest("03-aggregation/README.Rmd")
```

\clearpage