Control memory and core usage?

[lucygarner]

Hi,

Is there a way to control how many cores and how much memory GenerateCellHashingCalls uses. It is crashing on our linux server due to an out-of-memory error.

Best wishes,
Lucy

[bbimber]

Hello - I believe the default should be just one core. I'd need to refresh my memory on each specific tool though.

As far as memory, one of biggest users is actually tsne plotting. I'm not at my computer, but does that function expose an argument to suppress the tsne plots?

Also, about how large are these datasets that fail? How much memory are the jobs given? From the log, can you tell what it did last before it dies? Can you post the last couple lines of the output?

[lucygarner]

My code is as follows:

hashtag_counts <- as.matrix(GetAssayData(seurat_obj, slot = "counts"))
methods <- c("bff_cluster", "bff_raw", "gmm_demux", 
             "multiseq", "htodemux", "dropletutils")
set.seed(3245)
cell_hashing_calls <- GenerateCellHashingCalls(
    barcodeMatrix = hashtag_counts, 
    methods = methods,
    htodemux.positive.quantile = 0.99)

The function uses one core but the memory usage is high (see below). I am not sure exactly which functions are using the most memory (I would have to test separately), but it seems to be multiple since there are spikes in the memory usage.

I have ~60,000 cells in the largest sample.

Best wishes,
Lucy

[bbimber]

@lucygarner This is a little different from what i'm used to seeing, but can you try a couple things:

• Run this with one caller at a time, to see if one specific caller does this or no?
• My guess is tSNE might be to blame. You can disable this is a not-well-documented way. It currently needs to be done per-caller, so adding something like "htodemux.doTSNE = FALSE", "multiseq.doTSNE = FALSE", etc., will disable it for that caller.

The first one will tell us whether it's a specific algorithm, or whether it's something core to cellhashR. The second addresses the most likely culprit.

[lucygarner]

Hi @bbimber,

Thanks. I will disable tSNE calls as suggested above.

tSNE generation is crashing for my largest sample with the following error.

## [1] "Error generating tSNE, skipping"
## [1] "vector memory exhausted (limit reached?)"
## No traceback available

In another sample, I am getting the following error for all tSNEs:

## [1] "Error generating tSNE, skipping"
## [1] "long vectors not supported yet: complete_cases.c:192"
## No traceback available 

Best wishes,
Lucy