ArrayMemoryError when using tridesclous

[phanhuynh]

Hello team,
I've attached the error log for tridesclous.
The key phrase is:

raise self._exception\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 10.2 GiB for an array with shape (42923008, 64) and data type float32\n",
    "run_time": null

Any thoughts? Thank you very much!
spikeinterface_log.txt

[caniko]

The array is very large, I am not sure if that is normal.

You are out of memory (OOM), how much RAM do you have? You can increase your swap memory, but this will impact performance.

Unless there is a setting for PCs with low RAM on tridesclous, and there is no mistake in your configuration, the best way forward is to get a PC with more RAM.

[phanhuynh]

Hello Can, I'm not sure if my response came through via email (8 Sept):
Hi Can,
Thank you for your response.
However, the RAM on this computer is 128Go 3200 MHz DDR4 - 4x32Go ECC.
Seems like plenty. What do you think?

[alejoe91]

@samuelgarcia thoughts?

[caniko]

You can observe RAM usage with the task manager while the sorter runs, does it fill the RAM?

I would use my NVME drive as swap or re-design my data structures. Swap is really easy because it doesn't require much effort, and it can tell you how much memory the sorter really needs.

Did you try any other sorters? kilosort3?

Samuel knows best probably 💯

[phanhuynh]

Hi Can, thank you for the quick response.
Yes, this same data set worked with Kilosort3, Spyking Circus, and Iron Cluster.
I'll check if the ram is maxed out when I run it. I'll also loked into Swap. Thanks!

[phanhuynh]

Hello team!
I've observed the RAM. Nothing special happens but then it will max out to 100% right before crashing.
I've tried a few things from this stackoverflow page:

1. check that the python shell is 64bits instead of 32bits (it was correctly 64bits).
2. I've changed the overcommit handling, and changed the virtual memory limits. This didn't help (recall that I have 128GB Ram).

Now I'm considering changing the dtype attribute from float64 to uint8, but not sure if that's a paramater that I can change.

Any advice? Thank you!

[caniko]

I would advise against fp64 to uint8 as you would floor every decimal to an unsigned integer. You could try fp32 or even fp16, wouldn't go lower.

Note that your data will lose quality when deviating from the original to lower precision.

I would rather use a lot of swap. How much swap did you set? Could you set swap to 100-200 Gb, and see what happens? Try to use a fast drive.

[phanhuynh]

Hi Can:
If I understand correctly, swapping memory is what I did in #2 from my previous message. But correct me if I'm mistaken. Reference.
I've set my min = 192000MB (1.5XRAM) , max = 384000MB (3XRAM)

Overall, I'm surprised that I need to do this. I have a 2 TB SSD hard drive, plus 128 GB RAM. What are the minimum requirements for Tridesclous?

[caniko]

Tridesclous, and other spikesorters do have minimum requirements; however, the requirements are also affected by your own data. As I mentioned earlier, your data seems to be quite different from the usual. The data has the downstream effect of requiring larger np.ndarrays, which crashes the runtime because of system out of memory or OOM.

Also storage, 2TB, has no impact on this issue. This is strictly memory, which should be our focus. You have a very nice workstation, but if there are no issues on the tridesclous side it might be that you need even more RAM because of your data. There might be a bug, but we have to track it down.

Debugging points (try in descending order):

• How large is your raw data?
• Do you see the virtual memory in task manager? Total memory in task manager should be 384Gb after setting SWAP. Your description seems like you have been succesfull, making sure.
• Actually I am not certain you actually do experience OOM when setting your overall memroy to 384Gb. Did you see the entirity of 384Gb being filled up? I would start leaning towards the possibility of a bug if this is the case, because this is a lot of memory use.
• I would like to see what happens if you set your dtype to np.float32, I don't know if tridesclous has this feature. Shouldn't be too hard to implement.
• Can you try to run the spike sorting using a memory profiler like this one? So you run the sorter function inside the profiler by wrapping it:

from memory_profiler import profile

wrapped_run_tridesclous = profile(spikeinterface.run_tridesclous)
wrapped_run_tridesclous(**your_kwargs)

• You could also try to setup your environment inside WSL, and see if that helps. It is an adaptation of the Linux kernel to allow it to run on top of Windows OSs. This is quite advanced, I would try this after trying everything else.

[samuelgarcia]

Hi,
I am sorry for so long delay.
Normally tridesclous is doing chunk processing. So normally it cusums very few ram (depending chunk size of course).

Norammly the input is memmaped (using np.memmap) so normally we can adress very very dataset with a few ram.
The OS should load onde mand every chunk in memory and clean it when not necessary anymore.

Normally tridesclous should not triggered swap at all. I don't have swap on my own machine, I hate that I prefer true memory errore!

There must be a bug somewhere if tridesclous consum too much ram.
Could you running tridesclous with the docker ?

Unfortunatly tridesclous is very weel maintain since 2 years because my effort are now in spikeinterface.
I am working now on tridesclous2 which will be spikeinterface base and normaly efficient in computation time and memory consumption.

Sorry not to able to help more.

[phanhuynh]

Hi Sam - no worries, I know you're super busy. Great idea though about using docker. Unfortunately, I get the error pasted below.
@can, thank you for all your ideas! I'll try them if I can't get docker to work. But to answer your first question, the raw file is 20.6GB.

Starting container
Installing spikeinterface==0.95.1 in spikeinterface/tridesclous-base
Installing extra requirements: ['neo']
Running tridesclous sorter inside spikeinterface/tridesclous-base
Stopping container
---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Input In [9], in <cell line: 17>()
      1 #####################################################
      2 ### Option 1: Run local sorter (recommended) ###########
      3 #####################################################
   (...)
     14 ### Option 2: run sorter directly from docker ###
     15 #####################################################
     16 start_time = time.time()
---> 17 sorting_TDC = ss.run_sorter(
     18     'tridesclous',
     19     recording_saved,
     20     output_folder= '_output_TDC',
     21     remove_existing_folder=True,
     22     detect_threshold=threshold,
     23     docker_image="spikeinterface/tridesclous-base", 
     24     verbose=True,
     25 )
     26 print("--- _output_TDC took %s seconds ---" % (time.time() - start_time))
     27 print(f'Tridesclous found {len(sorting_TDC.get_unit_ids())} units')

File ~\Anaconda3\envs\sienv\lib\site-packages\spikeinterface\sorters\runsorter.py:136, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params)
    134         else:
    135             container_image = singularity_image
--> 136     return run_sorter_container(
    137         container_image=container_image,
    138         mode=mode,
    139         **common_kwargs,
    140     )
    142 return run_sorter_local(**common_kwargs)

File ~\Anaconda3\envs\sienv\lib\site-packages\spikeinterface\sorters\runsorter.py:548, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, extra_requirements, **sorter_params)
    546 if run_error:
    547     if raise_error:
--> 548         raise SpikeSortingError(
    549             f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    550 else:
    551     if with_output:

SpikeSortingError: Spike sorting in docker failed with the following error:
b'Traceback (most recent call last):\n  File "/Users/huynh/spike_epsztein/Phan/proj_nexus_v2/in_container_sorter_script.py", line 17, in <module>\n    sorting = run_sorter_local(\n  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py", line 157, in run_sorter_local\n    output_folder = SorterClass.initialize_folder(\n  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 123, in initialize_folder\n    shutil.rmtree(str(output_folder))\n  File "/usr/local/lib/python3.8/shutil.py", line 718, in rmtree\n    _rmtree_safe_fd(fd, path, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd\n    _rmtree_safe_fd(dirfd, fullname, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd\n    _rmtree_safe_fd(dirfd, fullname, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd\n    onerror(os.unlink, fullname, sys.exc_info())\n  File "/usr/local/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd\n    os.unlink(entry.name, dir_fd=topfd)\nPermissionError: [Errno 13] Permission denied: \'all_peaks.raw\'\n'

[caniko]

Running on docker and it still failing would eliminate the possibility of wrongful installation.

You can also try my repackeged version of spikeinterface with trideclous pip install spikeinterface-poetry[tridesclous] (this is supported only by myself), this is outside conda. You can use conda to create a virtualenv for you, and pip install inside.

I didn't mention it because it seems like you are running the sorter succesfully already, low chance of wrongful install.

Wish you the best of luck!

[phanhuynh]

Hi Can, Thank you for your response. However, the RAM on this computer is 128Go 3200 MHz DDR4 - 4x32Go ECC. Seems like plenty. What do you think?

…

On Thu, Sep 8, 2022 at 1:47 PM Can H. Tartanoglu ***@***.***> wrote: Seems like you are out of memory (OOM), how much RAM do you have? You can also increase your swap memory <https://superuser.com/questions/793304/how-to-increase-swap-memory-in-windows>, but this will impact performance. Unless there is a setting for PCs with low RAM, the best thing to do is to get a PC with more RAM. — Reply to this email directly, view it on GitHub <#932 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANCGACW4D66Z2IU2ISTSZMLV5HG5DANCNFSM6AAAAAAQHS6F4M> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- -Phan Huynh +33 7 66 23 36 98

[caniko]

Yes, it is plenty; however, all it takes for OOM is either bugged code and/or loading too much into memory. Hope this was made clear earlier.