Sigmoid aprox with tanh + fix sigmoid #3205
Conversation
gmuraru
changed the title
| else: | ||
| result = (1 + (self * -1).exp()).inverse() | ||
| one = self * 0 + 1 | ||
| result = one / (1 + (self * -1).exp()) |
There was a problem hiding this comment.
So basically you're suggesting .inverse() doesn't work ? :(
(The benefit of .inverse is that it's supposed to be faster)
There was a problem hiding this comment.
But yeah you're right it's failiing apparently sometimes, I was discussing of this two weeks ago on the call with André
gmuraru
changed the title
gmuraru
changed the title
gmuraru
force-pushed
the
gm-smpc-fix
branch
3 times, most recently
from
5d9f6c0
to
5b302ef
Compare
| @@ -560,6 +607,7 @@ def _tanh_chebyshev(tensor, maxval: int = 6, terms: int = 32): | |||
| where c_i is the ith Chebyshev series coefficient and P_i is ith polynomial. | |||
| The approximation is truncated to +/-1 outside [-maxval, maxval]. | |||
| Args: | |||
| tensor (tensor): values where the tanh needs to be approximated | |||
There was a problem hiding this comment.
typo this should be here :)
There was a problem hiding this comment.
Hmm..this is for _tanh_chebyshev (forgot to put it in the previous PR)
test/torch/tensors/test_precision.py
Outdated
| @@ -449,6 +449,7 @@ def test_torch_sigmoid_approx(workers): | |||
| alice, bob, james = workers["alice"], workers["bob"], workers["james"] | |||
|
|
|||
| fix_prec_tolerance_by_method = { | |||
| "chebyshev": {3: 6 / 100, 4: 1 / 100, 5: 1 / 100}, | |||
There was a problem hiding this comment.
To improve test speed could we remove the case precision=5 everywhere where it occurs? From what I see =4 is already compelling enough :)
| Approximates the sigmoid function | ||
| Compute the sigmoid using the exp approximation | ||
| sigmoid(x) = 1 / (1 + exp(-x)) | ||
| sigmoid(x) = (sigmoid(|x|) - 0.5) * sign(x) + 0.5 |
There was a problem hiding this comment.
Detail a bit the method for people which don't know about numerical stability etc
| @@ -449,6 +449,10 @@ def matmul(self, *args, **kwargs): | |||
| __matmul__ = matmul | |||
| mm = matmul | |||
|
|
|||
| def reciprocal(self): | |||
| ones = self * 0 + 1 | |||
| return ones / self | |||
There was a problem hiding this comment.
Let's keep this like this, until I come back to trying my approx inverse trick
gmuraru
force-pushed
the
gm-smpc-fix
branch
2 times, most recently
from
fccba78
to
11aaece
Compare
gmuraru
changed the title
gmuraru
force-pushed
the
gm-smpc-fix
branch
3 times, most recently
from
8e732fe
to
5f0d47d
Compare
Codecov Report
@@ Coverage Diff @@
## master #3205 +/- ##
==========================================
- Coverage 99.30% 94.55% -4.75%
==========================================
Files 4 146 +142
Lines 1866 15899 +14033
==========================================
+ Hits 1853 15034 +13181
- Misses 13 865 +852
|
Fixes #3131
Also adds the approximation for sigmoid using the tanh.