Give clearer guidance about multithreading in PyTorch, and how to disable it

[ezyang]

e.g. #16894

There's a lot of people asking about this, but no canonical source of information about it.

cc @jlin27 @mruberry

[vadimkantorov]

I think it would be nice to have a smart switch for limiting the number of CPU threads (activated from code and by a new torch-consumed env variable) that would set all the needed environment variables for OpenMP and known BLAS implementations.

It's important for the usecase of academic clusters managed by old deprecated software where a machine can be shared by several people and no rock-solid CPU-time/thread quotas exist. E.g. multiple data-loading threads and BLAS/OpenMP threads contribute to the CPU-affinity and CPU-cache thrashing issues. A single switch or guidance will be very helpful. An easy default way allowing to have good CPU-affinity without manual generating a mask and using the Linux command would also be very helpful.

[soumith]

@vadimkantorov as of v1.0.1, torch.set_num_threads(1) covers everything, including the BLAS libraries.

[vadimkantorov]

@soumith CPU affinity is the next step :)

[fisakhan]

@soumith @vadimkantorov using torch.set_num_threads(1) in Python I get in total 3 threads, where 1 thread is generated by Python. I use os.system( "top -H -b -n1 | grep python | wc -l") command to count the number of threads. Why do I get 2 (rather than 1) additional threads?
Can I make the total number of threads to 2 or 1 (including thread for python)?

[WillemvJ]

I am calling into PyTorch from c++. I have:

	at::set_num_interop_threads(1);
	at::set_num_threads(1);
	torch::set_num_threads(1);

Still, when I start inferencing a (small) neural network (with nogradguard) on CPU, I see 16 out of 128 processors which are using 100% capacity. Shouldn't torch::set_num_threads(1) ensure that only one thread is used?