`tanh` doesnt use all cores

[hughperkins]

When I run this script:

import torch

a = torch.rand(1000, 10000)
while True:
    print('.')
    a.tanh_()

and then open htop, I expect to see all 8 cores running at 100%, but only 4 seem to be running? :

In addition, those cores that are running, are only running at ~30-40%.

(Note that I'm not submitting a fix for this, just flagging it)

[jekbradbury]

What happens when you fiddle around with the pertinent environment variables (some subset of OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, and OPENMP_NUM_THREADS)?

[hughperkins]

I think this should be handled automatically really? At least, I dont see anything in any of the doc I've read suggesting one needs to do this. So, either the doc needs to be updated, or else this should be handled automatically, I reckon.

[hughperkins]

That said, with MKL_NUM_THREADS=8, no change:

OPENBLAS_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENMP_NUM_THREADS=8 python test_tanh.py: no change:

[hughperkins]

(tanh shouldnt be using blas? I would think it uses some combination of SSE and OpenMP?)

[jekbradbury]

Oh of course, sorry. Yeah, this is a good question then!

[apaszke]

It seems that Tanh doesn't use OpenMP at all. It calls into TH, and it uses TH_TENSOR_APPLY_2 which is single-threaded. TH could use some multi-core optimizations, but we don't have enough hands to do it right now.

We could provide some guidance if anyone wants to take a stab. We already have macros that extend TH_TENSOR_APPLY to make use of all cores in the contiguous case, so at least this improvement should be easy to add.

[ruotianluo]

I submitted a PR, check if it looks like what's expected. @apaszke

[fmassa]

If one wants to use OMP on TH_TENSOR_APPLY2, one could look into improving on top of torch/torch7#395 . But I think that having better support for AVX/SSE would be best.

[weiyangfb]

Closing since this has been fixed vix #2792