speed up rule save_counts (bottleneck)

[sreichl]

fwrite

• test
  • input has to be data.frame
  • changed for now in save_counts.R
• change in utils.R
• change in metadata_plot.R
• sctransform_cellScore.R

fread

• change in metadata_plot.R
• merge.R
• prepare.R

general

• make mr.pareto issue: look for write.csv across all MR.PARETO modules

https://rdrr.io/cran/data.table/man/fwrite.html

library(data.table)
fwrite(as.data.frame(GetAssayData(object = seurat_object, slot = "scale.data", assay = "SCT")), file = file.path(result_dir, paste0(step, 'scaled_', 'RNA', '.csv')), row.names=TRUE)

# more general
#fast writing
fwrite(as.data.frame(df), file=file.path("path/to/file.csv"), row.names=TRUE)

#fast reading
df <- data.frame(fread(file.path("path/to/file.csv"), header=TRUE), row.names=1)

[sreichl]

not the same! differences

• quotes -> fine/fixed
• column names differ '-' vs '.' -> check in metadata -> has to match for downstream tasks -> in metadata its '-'
• last row -> no gene name but rownumber?! -> fixed

new comparison
diff --brief <(tail -n +2 NORMALIZED_RNA.csv | cut -d, -f2-) <(tail -n +2 NORMALIZED_RNA_original.csv | cut -d, -f2-)
Files /dev/fd/63 and /dev/fd/62 differ

compare with brief diff
diff --brief NORMALIZED_RNA_original.csv NORMALIZED_RNA.csv

head -n 1 NORMALIZED_RNA.csv | cut -c 1-100
"",EMICROP_A_AAACCTGAGAATAGGG-1,EMICROP_A_AAACCTGAGACAATAC-1,EMICROP_A_AAACCTGAGACGACGT-1,EMICROP_A_

head -n 1 NORMALIZED_RNA_original.csv | cut -c 1-100
"","EMICROP_A_AAACCTGAGAATAGGG.1","EMICROP_A_AAACCTGAGACAATAC.1","EMICROP_A_AAACCTGAGACGACGT.1","EMI

tail -n 1 NORMALIZED_RNA.csv | cut -c 1-100
"28441",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,

tail -n 1 NORMALIZED_RNA_original.csv | cut -c 1-100
"AC007325.4",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0