Skip to content

Commit

Permalink
Implement Tanh Gelu Approximation (#61439)
Browse files Browse the repository at this point in the history
Summary:
1. Implements pytorch/pytorch#39853
2. Adds approximate boolean flag to Gelu
3. Enables Tanh Gelu approximation
4. Adds double backward support for Gelu
5. Enable Tanh Gelu in NvFuser

```
def gelu(x, approximate : str = 'none'):
    if approximate == 'tanh':
        # sqrt(2/pi) = 0.7978845608028654
        return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0))))
    else:
        return x * normcdf(x)
```

Linking XLA PR - pytorch/xla#3039

Pull Request resolved: pytorch/pytorch#61439

Reviewed By: mikaylagawarecki

Differential Revision: D33744717

Pulled By: jbschlosser

fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187
  • Loading branch information
rdspring1 authored and cpuhrsch committed Feb 2, 2022
1 parent 4e21fd6 commit 0f17137
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions nestedtensor/csrc/activation.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ namespace F = torch::nn::functional;

namespace at {

Tensor NestedTensor_gelu(const Tensor& self) {
Tensor NestedTensor_gelu(const Tensor& self, const int64_t approximate) {
if (is_nested_tensor_impl(self) && get_is_contiguous(self)) {
return wrap_buffer(
at::gelu(get_buffer(self)),
get_efficient_nested_size(self),
get_efficient_nested_stride(self));
}
return map_nested_tensor(
[](at::Tensor tensor) { return at::gelu(tensor); }, self);
[&approximate](at::Tensor tensor) { return at::gelu(tensor, approximate); }, self);
}

Tensor NestedTensor_elu(const Tensor& self, const Scalar& alpha, const Scalar& scale, const Scalar& input_scale) {
Expand Down

0 comments on commit 0f17137

Please sign in to comment.