Skip to content

Commit

Permalink
Removed the unnecessary softplus in _address_memory
Browse files Browse the repository at this point in the history
Removed the softplus in the softmax:
```python
        s = F.softmax(F.softplus(s), dim=1)
```
softmax already constrains the values to (0, +inf), the softplus doesn't achieve much.
  • Loading branch information
Jules Gagnon-Marchand authored and loudinthecloud committed Mar 27, 2018
1 parent db56fb8 commit d7b3840
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion ntm/head.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def _address_memory(self, k, β, g, s, γ, w_prev):
k = k.clone()
β = F.softplus(β)
g = F.sigmoid(g)
s = F.softmax(F.softplus(s), dim=1)
s = F.softmax(s, dim=1)
γ = 1 + F.softplus(γ)

w = self.memory.address(k, β, g, s, γ, w_prev)
Expand Down

0 comments on commit d7b3840

Please sign in to comment.