Skip to content
This repository has been archived by the owner on Apr 25, 2023. It is now read-only.

Non linearity after attention #6

Merged
merged 3 commits into from Aug 28, 2018
Merged

Non linearity after attention #6

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
44e32fe
adds non-linearity to attn weights
pskrunner14 Aug 28, 2018
91dcbd7
adds non-linearity to attn weights
pskrunner14 Aug 28, 2018
acb9e26
Delete settings.json
pskrunner14 Aug 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions model.py
View file
Open in desktop
Expand Up @@ -40,11 +40,11 @@ def forward(self, hidden, encoder_outputs):
h = hidden.repeat(timestep, 1, 1).transpose(0, 1)
encoder_outputs = encoder_outputs.transpose(0, 1) # [B*T*H]
attn_energies = self.score(h, encoder_outputs)
return F.softmax(attn_energies, dim=1).unsqueeze(1)
return F.relu(attn_energies, dim=1).unsqueeze(1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And to leave this line alone


def score(self, hidden, encoder_outputs):
# [B*T*2H]->[B*T*H]
energy = self.attn(torch.cat([hidden, encoder_outputs], 2))
energy = F.softmax(self.attn(torch.cat([hidden, encoder_outputs], 2)))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you need a ReLU here.

energy = energy.transpose(1, 2) # [B*H*T]
v = self.v.repeat(encoder_outputs.size(0), 1).unsqueeze(1) # [B*1*H]
energy = torch.bmm(v, energy) # [B*1*T]
Expand Down