Handle automatic chunks duration for SC2

[yger]

When using large number of cores, we might want to automatically set the chunk_size as function of the available RAM and/or number of cores. While I know that currently, in SI, the preprocessing nodes can consumes more RAM than chunk_size, at least this will avoid swapping and automatically adjust the sizes

[zm711]

Couple more cosmetic questions.

[zm711]

are these prints or warnings? in general.

[yger]

I'll use the warning package instead

[zm711]

same for these prints.

[yger]

same here

[zm711]

I think you need to say what units of RAM this will work with. I'm confused by this with the way it is currently written. I think users will need a bit more info to really understand how to use this.

[b-grimaud]

With a few modifications, it can process a minute long recording in 8h45.

• In the main sorting function, it would often crash at estimate_templates. I added a call to get_optimal_n_jobs with the same ram_requested as in clustering/circus.py.
• In cases where the requested RAM is high and/or the memory limit is low, int will round n_jobs down to 0, which will return an error. I think get_optimal_n_jobs should ensure that n_jobs is at least 1, or at least return a specific warning.
• In clustering/circus.py, I also had crashes at remove_duplicates_via_matching. Passing the same job_kwargs as those used in estimate_templates does the trick, but another call to get_optimal_n_jobs with a proper estimate of the requested amount of RAM could be more appropriate.
• In clustering_tools.py, detect_mixtures sets n_jobs to 1 in its main loop. I'm guessing this part can't be parallelized ?

[yger]

Ok, good to know, I'll update the PR. Meanwhile, I'll also make

• a PR to bypass the estimate_templates() call needed at the end of the clustering. In fact, templates could be inferred (at the cost of slight smoothing) from the SVD components that have been precomputed and are saved in RAM.
• a PR to use the new graph-based clustering that we developed with sam. Such a clustering, taking the whole probe into account as a whole, would not need to deal with duplicates and thus detect_mixtures() should not be needed anymore. That's being said, I think that this function could also be skipped in your case (and in recent version of circus2). Because the final merging has been improved, keeping some duplicates should not hurt too much, and you are welcome to give it a try. How many are removed by such a function in your case?

Thanks again for the feedback

[b-grimaud]

Thanks for the update !

detect_mixtures removes 6 units out of 3705, so about 0,2%. This seems entirely acceptable.