This is (or will be) a Dynamic Statistical Comparison to estimating (or testing) the "log-fold-change" in mean between two groups from count data.
Our intention is to initially focus on data from single-cell experiments.
The goal is to compare methods for estimating the log-fold-change between two groups.
Methods will input:
- $Y1 a numeric matrix of data from group 1 (p by n, p genes in columns, n samples/cells in rows)
- $Y2 a numeric matrix of data from group 2 (p by n, p genes in columns, n samples/cells in rows)
- $X1 an n vector of covariates from group 1 (eg "library size")
- $X2 an n vector of covariates from group 2
Methods will output:
- $log_fold_change_est a vector of estimates of log(mu1/mu2) for each gene where mu1 is the mean of group 1 and mu2 is the mean of group 2
- $s_hat a vector of standard error for the estimated $log_fold_change
- $p a p-vector of p values testing whether each log-fold change is 0
We will create synthetic data (from real data) that have known log-fold-change values, and compare the estimates with the real values. We will also assess calibration of p values (eg on null data, we should get uniform p values) and power.
To create data we will take a file containing count data and select samples at random to create two groups. These will be "null" data.
- file of data
- n1 sample size for group 1
- n2 sample size for group 2
- p number of genes
- pi0 proportion of nulls
- g a distribution on non-zero effects
- $log-fold-change (true value of log_fold_change for each gene)
List methods we might want to use...
We manage package installation using conda. Aftering installing conda, you have two options for installing the dependencies:
- Create a new environment with
conda env create --file environment.yaml source activate dsc-log-fold-change
- Directory install with
conda config --add channels jdblischak conda config --add channels defaults conda config --add channels bioconda conda config --add channels conda-forge conda install --file requirements/conda-forge \ --file requirements/bioconda \ --file requirements/jdblischak
The advantage of the first option is convenience. It creates an environment and installs packages from specific channels, all in one step. The advantage of the second step is that it is faster and more robust.
If you need to add a new package for the benchmark, please add it to both
environment.yaml and to one of the files in
The main DSC file is
benchmarks.dsc. To see what is available:
and to run the benchmark:
Or to run a minimal test benchmark, eg
./benchmark.dsc --target "get_data * wilcoxon_test" --truncate --replicate 1 # default is in fact --replicate 1