High memory usage after loading saved joined Seurat file

[MGP1893]

Hello,
I've only been using Seurat for a couple of months, so my technical knowledge is still developing. Recently I tried to load a saved Seurat object, 9.9GB, and found that memory usage after the file was fully loaded was quite high, to the point where I was getting BadAlloc warnings while doing simple things. I tried gc() after that several times, but there was no improvement in memory usage. If I run through the code, however, to create the same file, garbage collection works fine. Hopefully this makes sense to someone. Is there any other way to force garbage collection? (I also tried the "Free Unused R memory" button in the environment tab.

[Anjaney10]

Hi @MGP1893, even I am new to Seurat, but can you tell what was the saving format of the seurat object? I suppose likely RDS? I usually save it as .qs, I think that is faster. Also, you might be hitting the RAM capacity of your system. It will be great if you can specify what system are you using and how much capacity it has.

[MGP1893]

Hello,
Indeed the file is saved as RDS. I can definitely try saving as the other format.
I am on Windows and have 64G of RAM and I have increased the page file so that it can access 133 GB. When I load in the saved file (says 20.4 GB where it is saved), it takes several minutes to load and then appears to be successful but then the memory usage stays up over 80% no matter how long I give it or if I give it a gc() command. When I run through the code to get the same file, my Large Seurat object is 9.9 GB and I can garbage collect memory usage down to 20%.

[Anjaney10]

@MGP1893 Is your loaded object just the Seurat object (which is 9.9 GB), or did you save other environment variables in that RDS object? gc() won't help unless you remove some objects, afaik.

[igrabski]

Hi, in addition to Anjaney10's question above (whether other environment variables are saved), can you verify whether re-saving the file after running through the code, and then re-loading it reproduces this problem or not?

[MGP1893]

Hello,
Afraid this slipped my mind as I've been attempting SCENIC for the first time. I will probably have to attempt this next week. As far as Anjaney's question above, I have commands in the code to remove the files that come before the joined Seurat after joining just to save space. If I run the code to get to that place, the only thing in the environment under Data is one Large Seurat object.
Next week, I will run through the code, save it again, see if the problem is reproduced, and take screen shots.
Thanks to both of you!

[MGP1893]

Ok so I've run this over and over both ways- saving a file and then opening it after restarting R, and running the code to get to the file. It seems like the point at which it crashes is never the same if I open a saved file, but it's going to eventually crash somewhere. I am wondering if the amount of memory committed has something to do with it.

Here are some screenshots after running the code and saving a file. This one I can give time and gc() and memory usage will drop down.

Here are some after loading the same saved file. This one seems to maintain a very high level of committed memory regardless of what I do.

What I mean by committed memory. I'm running something else right now so this isn't the best example, but if I load a saved file, it will maintain like 80% committed.

Hopefully I'm just doing something dumb and you guys can point me in the right direction. I'm running Windows, I have 64G + a page file, and I'm saving files onto an external hard drive.

[samuel-marsh]

Hi,

couple of notes that may be of help. First, saved file size and loaded in memory file size are not going to be equivalent often due to compression. On that note I would also recommend format like qs2 or RDS vs RData for saving files as they will be significantly faster read/write and smaller saved file size.

Second, have you run DietSeurat to remove unneeded info from object?

Third, if memory usage is causing the crash I would recommend looking into using BPCells based Seurat object vs loaded in memory.

Best,
Sam