[BUG] Loading Expression CSV Likely Causes Memory Overflow

[jasonycin]

Loading Issue Causes Memory Overflow

Problem

We typically run OncoProfilingTools on VCU's high-performance research cluster which has immense memory and CPU capabailities. Therefore, we have been able to run and load datasets without an issue.

We have noticed that when we load the DepMap Expression matrix (OmicsExpressionTranscriptTPMLogp1HumanAllGenes.csv) on a local machine (i.e. Jason's macbook) then we run into memory issues, even with 32 GB of available memory.

The file is approx. 2.9GB and should fit well into memory. In this case, we need to just optimize the loading and make it more efficient. We expect users to be able to use this library on their local machines without access to a research cluster.

Steps to Reproduce

library(OncoProfilingTools)

exp <- OncoExperiment()
exp <- load_assays(exp, AssayTypes$Expression, path = "/folder) # you will need the file in a folder on your machine

Tips to Solve

Each assay type has a different loader function that is dynamically acquired based on the filename.
We are using the following library to load it in a data format easy to work with:

OncoProfilingTools/R/method-load-expression-assay.R

Line 5 in 49f8d42

library(data.table)

• Decide whether this library is good or not. If you stick with it, optimize it. Define what the data types of columns are. Ensure few or no copies of the entire matrix are made during loading. Do we need a whole different approach such as chunking? Imagine a user only has 8GB of memory. I think it is reasonable to expect >=16GB.
• Check use of the S4Vectors and ensure they're not duplicating large columns either.
• In the very end, we still need to return the same class and be able to create a SummarizedExperiment
  

  OncoProfilingTools/R/method-load-expression-assay.R

  Lines 399 to 431 in 49f8d42

  	## ---------------------------------------------------
  	## Return ExpressionAssay
  	## ---------------------------------------------------
  	assay_metadata <- list(
  	assay_name = assay_name,
  	source = "DepMap",
  	source_file = normalizePath(path, winslash = "/", mustWork = TRUE),
  	unit = unit,
  	normalized = normalized,
  	feature_type = feature_type,
  	id_col = id_col,
  	filtered_to_default_model_entries = "IsDefaultEntryForModel" %in% colnames(expression_index),
  	n_rows_before_default_filter = n_rows_before_default_filter,
  	n_rows_after_default_filter = n_rows_after_default_filter,
  	model_metadata_loaded = !is.null(model_metadata_path),
  	model_metadata_file = if (!is.null(model_metadata_path)) {
  	normalizePath(model_metadata_path, winslash = "/", mustWork = TRUE)
  	} else {
  	NA_character_
  	}
  	)
  	se <- SummarizedExperiment::SummarizedExperiment(
  	assays = S4Vectors::SimpleList(
  	expression = expression_matrix
  	),
  	rowData = feature_metadata,
  	colData = model_metadata,
  	metadata = assay_metadata
  	)
  	methods::as(se, "ExpressionAssay")

• Adjust the the other loading functions to match.