[z_Deprecated] Benchmarking methods

2.1 For methods implemented in R

For methods implemented in R including combat (combat/combat-seq), limma, MMUPHin, ConQuR (ConQuR/ConQuR_libsize/ConQuR_rel), they can be run on a dataset along the preprocessing steps a dataset needs prior to running each of these methods using the scripts /benchmark/methods_benchmarking.R for the real-world datasets and /benchmark/methods_benchmarking_sim.R for simulation datasets.

Note that if you run into file path related errors, please check to ensure all the requested files in methods_benchmarking_sim.R and methods_benchmarking.R exist.

2.1.1 Running these methods for simulation

To run the methods in scale in slurm, relevant parallelization and iterative running has been set up in the script methods_benchmarking_sim.R. For example, one can run the following to run all data on iteration 1 of the simulated dataset for running methods and generating inntegrated datasets for the minimally viable parameter set:

• First, in command line, enter the benchmark folder if not already and enter R:

cd benchmark
R

• Then, run the following script

source('./methods_benchmarking_sim.R')
overall_path = './trial/simulate'
output_dir = './trial/simulation_data_output_count_norelation' # pls make sure this dir exists
or_l = c(1.25)
cond_effect_val_l = c(0.5)
batch_effect_val_l = c(0.5)
method_l = c("combat_seq", "limma", "MMUPHin", 'ConQuR', 'ConQuR_libsize')
scaled_slurm_methods_bencharking(output_dir, overall_path, method_l, or_l, cond_effect_val_l, batch_effect_val_l, GLOBAL_ITER, count = TRUE)

The results of running such scripts are exemplified in this folder, one folder for each dataset. Note that for the methods implemented in python (2.2), those files will be generated with code in that section.

Note that the script has been written in a way that makes it easy to call and run on different iterations of input, as long as you uncomment the last commented code chunk in the methods_benchmarking_sim.R. For example, running Rscript methods_benchmarking_sim.R 10 will run iteration 10. To scale run simulation in slurm, one can reference file /benchmark/slurm_bash_scripts/step1_methods_run_sim_batch.sh and move the revised file back to /benchmark before running on slurm in scale.

2.1.2 Running these methods for real-world datasets

Similarly to running these methods for simulation, you can run the methods for the real-world dataset by calling the script like below after running R in the benchmark folder:

source('./methods_benchmarking.R')
# autism 2 microbiomeHD
current_root = '../data/cleaned_data/autism_2_microbiomeHD/autism_2_microbiomeHD'
output_root = './trial/rw_methods_out/autism_2_microbiomeHD/autism_2_microbiomeHD'
run_methods(paste0(current_root, "_count_data.csv"),
    paste0(current_root, "_meta_data.csv"),
    output_root,
    dataset = "Dataset",
    batch_ref = 'asd_son',
    covar = c("DiseaseState"),
    count = TRUE,
    used_methods =  c("combat_seq", "limma", "MMUPHin", 'ConQuR', 'ConQuR_libsize')
)

To simplify the running of methods on each of the four real-world dataset, one can simply run the following in command line.

Rscript methods_benchmarking.R 1 # 1 for autism_2_microbiomeHD, 2 for cdi_3_microbiomeHD, 3 for ibd_3_CMD, 4 for crc_8_CMD

The results of running such scripts are exemplified in this folder, one folder for each dataset. Note that for the methods implemented in python (2.2), those files will be generated with code in that section.

And of course, you can run this in scale by referencing /benchmark/slurm_bash_scripts/step1_methods_run_rw.sh and move the revised file back to /benchmark for slurm runs.

2.2 For methods implemented in Python

2.2.1 Simulation

For methods implemented in Python including harmony, percentile_normalization, they can be run on a dataset along the preprocessing steps a dataset needs prior to running each of these methods using the script /benchmark/evaluate.py -o 1. Similarly to before, one can run this option in both an iterative python kernel (iPython) or through command line.

In iPython, once you enter the benchmark directory, you may do the following for simulation as a minimum viable example:

from evaluate import *
overall_path = './trial' # pls make sure this dir exists
or_l = [1.25]
cond_effect_val_l = [0.5]
batch_effect_val_l = [0.5]
GLOBAL_DATATYPE = 'count'
related = 'no'
binarizing_agent_biovar = 'cond_1'
iterative_methods_running_evaluate(run_or_evaluate = 'run', datatype = GLOBAL_DATATYPE, iter = 1, or_l = or_l, cond_effect_val_l = cond_effect_val_l, batch_effect_val_l = batch_effect_val_l, 
                    address_XY_dir_path = './trial/simulate', 
                    output_dir_path = f'./trial/simulation_data_output_{GLOBAL_DATATYPE}_{related}relation', 
                    eval_dir_path = overall_path+f"/simulation_data_eval_{GLOBAL_DATATYPE}_{related}relation",
                    methods_list = methods_list_dict[GLOBAL_DATATYPE], 
                    binarizing_agent_biovar = binarizing_agent_biovar)

To explain the parameters, we first note that iterative_methods_running_evaluate is a generic wrapper function, where arguments

• run_or_evaluate, a str type argument, should be either run or evaluate. In this case, you should use run because we are running the two data integration methods implemented in python; for the later evaluation section, you use evaluate
• datatype, a str type argument, should be either count or relab.
• iter is simulation iteration for data generation (default=1 without user selection).
• or_l, cond_effect_val_l, batch_effect_val_l: three parameter lists to generate data for.
• address_XY_dir_path, a str type argument denoting the path of the directory where the original (uncorrected) count/relab dataset and the corresponding metadata (Y) should be.
• output_dir_path, a str type argument denoting the path of the directory where the corrected/integrated datasets after running the methods and the related statistics should be.
• eval_dir_path, a str type argument denoting the path of the directory where the evaluation results after analyzing the characteristics of each corrected dataset should be. (Need to be defined but not super relevant for this module)
• methods_list, a list of str denoting the methods whose integrated datasets are then evaluated. For example, it can be ["nobc", "harmony", "combat_seq", "limma", "MMUPHin", "ConQuR", "ConQuR_libsize", "percentile_norm"] for count datasets, and ["nobc", "harmony", "combat", "limma", "MMUPHin", "ConQuR_rel", "percentile_norm"] for relab sets. (Need to be defined but not super relevant for this module)
• binarizing_agent_biovar, a str type argument denoting the name of the column in the input metadata for the column of the binary biological condition variable. For the simulated dataset in our setup, it should be cond_1.

The results of running such scripts are exemplified in this folder, one folder for each dataset.

To make it easier for streamlining, we also defined the following:

• related, a str type argument, should be either yes or no denoting whether confounding exists or not between library size and the simulated cross-study batch effect.

Alternatively, we provide a command line option, which might require you to go in and change certain codes in evaluate.py, but allows you a more streamlined running experience:

1. Replace the parameter combination set up (line 1119-1122).
2. Ensure that the methods run and the paths where the input files are taken and output files generated are correct for the section after the line if args['option'] == 1:
3. Run python3 /benchmark/evaluate.py -o 1 -d relab -r yes -i 1 -a cond_1 -p /athena/linglab/scratch/chf4012. You can easily replace relab with count (the default) or yes with no (default), 1 with other iteration numbers, and the path with the overall path where all your files are generated. For details, please check the argparse section of the file evaluate.py by typing python3 evaluate.py -h.

For this alternative option, you can scale run with slurm by referencing the file /benchmark/slurm_bash_scripts/step1_python_methods_run_sim.sh, revising it, and moving it to benchmark folder to slurm run.

2.2.2 Real-world dataset

• For the iPython option, first enter ipython in command line in the benchmark folder

cd benchmark
ipython

• Then run the following

vars_use = ["Dataset"]
IDCol = 'Sam_id'
overall_path = './trial' # pls make sure this dir exists
address_X = '../data/cleaned_data/autism_2_microbiomeHD/autism_2_microbiomeHD_count_data.csv'
address_Y = '../data/cleaned_data/autism_2_microbiomeHD/autism_2_microbiomeHD_meta_data.csv'
data_mat, meta_data = load_results_from_benchmarked_methods(address_X, address_Y)
res_h, meta_data_h = generate_harmony_results(data_mat, meta_data, IDCol, vars_use, overall_path+"/rw_methods_out/autism_2_microbiomeHD/autism_2_microbiomeHD_harmony")
percentile_norm(address_X, address_Y, "DiseaseState", "ASD", "comma", overall_path+"/rw_methods_out/autism_2_microbiomeHD/autism_2_microbiomeHD")

One can scale run the methods on simulated dataset by referencing the lines containing -o 1 in the slurm file /benchmark/slurm_bash_scripts/evaluate_run_sim.sh and scale run in slurm.

For the ease of running, one can use option 4 in the evaluate.py script once they have checked to ensure all the variables and file paths are correct in the section after line elif args['option'] == 4: of the file evaluate.py.

The results of running such scripts are exemplified in this folder, one folder for each dataset.