add AVX2 implementation for sigmoid function

[vedanuj]

PR introduces AVX2 optimization for sigmoid floats. Issue #4929. The internal benchmark shows ~10x speedup.

• Added AVX2 vectorized sigmoid using the 8-way vectorized exp (exp256_ps) in avx_mathfun.h.

• Implemented vector dispatch for sigmoid. Since sigmoid function is defined for floats and doubles only, for now, added preprocessor #ifdef to init sigmoid dispatch only for float and double.

• Vector functions in THVector.h were not called for all of the basic functions in floating point or double only. Changed the LAB_IMPLEMENT_BASIC_FUNCTION define in THTensorMatch.c to use THVector_(NAME) implementations if the inputs are contiguous. For the functions that do not have vectorized SIMD implementations will use the same default function from THMath.h

Benchmark

Non-vectorized sigmoid :

In [1]: import torch
In [2]: x = torch.randn(10000,10000)
In [3]: %time _ = x.sigmoid()
CPU times: user 2.8 s, sys: 130 ms, total: 2.93 s
Wall time: 737 ms

In [1]: import torch
In [2]: x = torch.randn(1000,1000)
In [3]: %time _ = x.sigmoid()
CPU times: user 29.1 ms, sys: 4.16 ms, total: 33.3 ms
Wall time: 8.63 ms

AVX2 Vectorized sigmoid

In [1]: import torch
In [2]: x = torch.randn(10000,10000)
In [3]: %time _ = x.sigmoid()
CPU times: user 206 ms, sys: 106 ms, total: 312 ms
Wall time: 78.2 ms

In [14]: x = torch.randn(1000,1000)
In [15]: %time _ = x.sigmoid()
CPU times: user 179 µs, sys: 2.95 ms, total: 3.13 ms
Wall time: 858 µs

[vedanuj]

PR for Issue #4929
@zdevito

[soumith]

@pytorchbot add to whitelist

[zdevito]

This looks good! Let's see if we can track down what is going on with THVector_ dispatch to make sure we aren't degrading performance in some way for the non-vectorized functions by adding the THVector_ dispatch.

[vedanuj]

Changes in new commit :

• Added a new macro LAB_IMPLEMENT_VECTORIZED_FUNCTION for the vectorized basic functions. Currently only sigmoid uses this macro which redirects to the vectorized implementation.

• Although there is no significant improvement in performance(due to exponent being the computationally dominant operation), replaced _mm256_mul_ps(minus_one, YMM0) with _mm256_sub_ps(zero, YMM0) which should be less expensive.

(Please re-review the code @zdevito @soumith @fmassa)

[goldsborough]

@cpuhrsch do you want to take a look at this? I know you're working on CPU improvements.

[cpuhrsch]

@goldsborough I'm writing those directly within ATen native, so this code won't conflict with anything I'm writing right now.

[ezyang]

There is probably more to look into re this code path, but I don't see why we shouldn't take such an obvious improvement for sigmoid.