control/query thread limit

[ereming]

Hi, apologies if this is already addressed elsewhere. I'm running phy/kk2 on a SLURM computing cluster. I'd like to be able to know how many cores kk2 (and in the future, SpikeDetekt2) will be using so that I can make the appropriate resource request. Is that possible? Is it adjustable?

[thesamovar]

It uses OpenMP so you can set the OMP_NUM_THREADS environment variable to whatever number of cores you like (otherwise it will by default use the maximum number available).

[rossant]

SpikeDetekt will use a different mechanism: you'll use IPython.parallel to set up the engines, either on the same computer, or on different computers. It should also work with cloud architectures like Spark.

[nippoo]

Some hybrid of both will probably occur for KK2. I think the plan is to support the current shared-memory multithreading as well as distributed memory multiprocessing through IPython.parallel, but only the former is currently available.

As for the ideal optimum number of cores to use: this probably depends on the size of your dataset; we'd welcome feedback on what you've found works best for you!

[ereming]

This is probably my mistake, but I'm not having luck with the OMP_NUM_THREADS variable. I'm not sure if you guys are familiar with SLURM, but my script is along the lines of:

#SBATCH <params>
export OMP_NUM_THREADS=1
phy cluster-auto kwik_file.kwik

Which isn't successful in limiting phy to one thread.

[thesamovar]

HI,

Not your fault: KK2 ignores the OMP_NUM_THREADS argument. I have fixed
this in the current master version of KK2 by adding a new parameter
num_cpus which you can set to whatever value you like.

Dan

On 17/07/2015 20:32, ereming wrote:

This is probably my mistake, but I'm not having luck with the
|OMP_NUM_THREADS| variable. I'm not sure if you guys are familiar with
SLURM, but my script is along the lines of:

|#SBATCH |
|export OMP_NUM_THREADS=1|
|phy cluster-auto kwik_file.kwik|

Which isn't successful in limiting phy to one thread.

—
Reply to this email directly or view it on GitHub
#66 (comment).