Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
01-metric_characterisation
02-metric_conformity
03-aggregation
optimise_feature_importance
README.Rmd
README.md

README.md

Metrics for comparing two trajectories

Good metrics are crucial for an unbiased and comprehensive benchmarking. Here we test several possible metrics for comparing two trajectories.

Metric characterisation

A first characterisation of the metrics. For each metric we:

  • generate some examples to get some intuition on how the metric works
  • test the robustness to a metric to stochasticity or parameters when appropriate
# script/folder description
1 πŸ“„correlation.R Characterisation of the cordist
2 πŸ“„topology.R Characterisation of the Isomorphic, edgeflip and HIM
3 πŸ“„clustering.R Characterisation of the F1branches and F1milestones
4 πŸ“„featureimp.R Characterisation of the corfeatures and wcorfeatures

The results of this experiment are available here

Metric conformity

Differences between two datasets should be reflected in certain changes in the metrics. This can be formalised in a set of rules, for example:

  • If the position of some cells are different than in the reference, the score should decrease.
  • If the topology of the network is different than that in the reference, the score should not be perfect.
  • The more cells are filtered from the trajectory, the more the score should decrease.

Here, we assess whether metrics conforms such rules empirically:

# script/folder description
1 πŸ“„generate_reference_datasets.R Generation of toy datasets used as reference to assess metric conformity
2 πŸ“„evaluate_perturbations.R Perturbing the reference datasets (for example shuffling the cells) to check the effect of this perturbation on a metric
3 πŸ“„assess_conformity.R Using the scores on perturbed datasets, assess whether the metrics follow certain rules
4 πŸ“„assess_similarity.R Assess the similarity between metrics
πŸ“„helper-create_perturbation_images.R Helper for creating small images for each rule
πŸ“„helper-rules.R Helper functions containing the rules

The results of this experiment are available here.

Aggregation

Here, we create some examples here for why and how we aggregate the scores.

# script/folder description
1 πŸ“„normalisation_reasoning.R
2 πŸ“„aggregation_example.R

The results of this experiment are available here.

The results of this experiment are available here.

You can’t perform that action at this time.