Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attention Mechanism Implementation Issue #1472

Closed
zhzou2020 opened this issue Jan 15, 2016 · 25 comments
Closed

Attention Mechanism Implementation Issue #1472

zhzou2020 opened this issue Jan 15, 2016 · 25 comments

Comments

@zhzou2020
Copy link

The problem is that I have an output a from a LSTM layer with the shape of (batch,step,hidden), and an output b from another layer, which is called the weights(or attention), with the shape of (batch,step), and I have no idea how to do an operation which computes the weighted sum given the two outputs.
like this:

a0, a1, a2 = a.shape
b0, b1 = b.shape
for i in range(a0):
for j in range(a1):
for k in range(a2):
c[i][k]+=a[i][j][k]*b[i][j]

Can this be done in keras?

@jfsantos
Copy link
Contributor

Wouldn't a TimeDistributedMerge layer work for you?

@farizrahman4u
Copy link
Contributor

@zzjin13 Here you go..

from keras.layers.core import*
from keras.models import Sequential

input_dim = 32
hidden = 32

#The LSTM  model -  output_shape = (batch, step, hidden)
model1 = Sequential()
model1.add(LSTM(input_dim=input_dim, output_dim=hidden, input_length=step, return_sequences=True))

#The weight model  - actual output shape  = (batch, step)
# after reshape : output_shape = (batch, step,  hidden)
model2 = Sequential()
model2.add(Dense(input_dim=input_dim, output_dim=step))
model2.add(Activation('softmax')) # Learn a probability distribution over each  step.
#Reshape to match LSTM's output shape, so that we can do element-wise multiplication.
model2.add(RepeatVector(hidden))
model2.add(Permute(2, 1))

#The final model which gives the weighted sum:
model = Sequential()
model.add(Merge([model1, model2], 'mul'))  # Multiply each element with corresponding weight a[i][j][k] * b[i][j]
model.add(TimeDistributedMerge('sum')) # Sum the weighted elements.

model.compile(loss='mse', optimizer='sgd')

Hope it helps.

@zhzou2020
Copy link
Author

I found this code cannot work properly when the input of the LSTM is masked.
How can I solve it with masked input?

@philipperemy
Copy link

@zzjin13 Can you clarity? You pad with 0 and you give mask=True to all your LSTM layers? This is masking for you?
Because the logic is exactly the same, masked or not in this case.

@ylqfp
Copy link

ylqfp commented Apr 28, 2016

@farizrahman4u Thanks so much! I'll have a try.

@philipperemy
Copy link

philipperemy commented May 23, 2017

I've just written a very simple Hello world for attention with visualisations (with the new Keras syntax)

Have a look: https://github.com/philipperemy/keras-simple-attention-mechanism

It might help you :)

@abali96
Copy link

abali96 commented May 24, 2017

@philipperemy which form of attention is this? Is there a specific paper you referenced in developing your attention model? Thanks for open sourcing by the way!

@philipperemy
Copy link

philipperemy commented May 25, 2017

Thanks for your feedback! It's the basic attention mechanism when you derive a probability distribution over your time states with n-D time series (no encoder-decoder here).

@abali96 I didn't have any papers in mind when I implemented it.

But a good paper you can have a look at is this one:

@nnulcm
Copy link

nnulcm commented Jul 19, 2017

Can I have a pre-trained attention layer weights?Such as the probability distribution of each word?

@xiaoleihuang
Copy link

@philipperemy check out Bengio's paper
I think there are some missing implementations in your codes.

@philipperemy
Copy link

@xiaoleihuang this one is for Neural Machine Translation, basically sequence to sequence attention. My implementation does not deal with this.

@xiaoleihuang
Copy link

@philipperemy Hi, I am not sure what formula you are based on. According to the formulas in the paper, I mean I did not see two basic steps in your attention_3d_block function

  1. calculate the dot production of eij;
  2. the weight aij is not normalized;
  3. there is a multiplication step in your merge, but shall it follows with a sum?

@philipperemy
Copy link

@xiaoleihuang I didn't base my implementation on any known paper. My attention here is just a softmax mask inside the network. It basically gives you a normalized distribution of the importance of each time step (or unit) regarding an input.

Intrinsically, it should not help the model perform better but it should help the user understand which time steps contribute to the prediction of the model.

@xiaoleihuang
Copy link

Hi, @philipperemy I see. I can understand your intuition. You utilize the input and compute it as a kind of "weights" and it will automatically optimized by the Neural Network (Permute->Reshape->Dense). In order to do matrix multiplication, you repeat the output from Dense. But there is an issue with your implementation: the attention defined in the paper is a dot production, but yours is a vector. What theory would supports such an operation? I am a little confused. But I think yours is a good idea. Additionally, I found there might be some issue with you K.function part in get_activations.

@bicepjai
Copy link

bicepjai commented Sep 9, 2017

Is this issue/discussion related to #4962

@v1nc3nt27
Copy link

@xiaoleihuang There is a dot product inside the dense, isn't there? The permuted Input is multiplied with the weight matrix in the dense layer.

@philipperemy Did you manage to use your attention mechanism successfully in a real project? I've tested it, but the score doesn't really change and the word's highlighted are rather random from what I can see. Would be nice to see it working in a bigger context. By the way, what is the Reshape layer for?

@xu-song
Copy link

xu-song commented Jan 16, 2018

@philipperemy @farizrahman4u
The shape in farizrahman4u's case has some mistake. I revise it as the following

#The weight model  - actual output shape  = (batch, step)
# after reshape : output_shape = (batch, step,  hidden)
model2 = Sequential()                            # input_shape  = (batch, step, input_dim)
model2.add(Lambda(lambda x: K.mean(x, axis=2)))  # output_shape = (batch, step)
model2.add(Activation('softmax'))                # output_shape = (batch, step)
model2.add(RepeatVector(hidden))                 # output_shape = (batch, hidden, step)
model2.add(Permute(2, 1))                        # output_shape = (batch, step, hidden)

@Ashima16
Copy link

Hi @farizrahman4u, I am getting "init() takes 2 positional arguments but 3 were given" error in the above code mentioned by you..Could you please help, as I am new to all this.

@caugusta
Copy link

Hi @Ashima16, that usually means you've provided too many inputs to a function or method. Make sure, for example, that you're providing only model1 and model2?

@Ashima16
Copy link

Hi @caugusta, thanks for the response..

I tried executing the same code as mentioned above with step value=1..
I am getting the following error:


TypeError Traceback (most recent call last)
in ()
17 #Reshape to match LSTM's output shape, so that we can do element-wise multiplication.
18 model2.add(RepeatVector(hidden))
---> 19 model2.add(Permute(2, 1))
20
21 #The final model which gives the weighted sum:

TypeError: init() takes 2 positional arguments but 3 were given

@caugusta
Copy link

Hi @Ashima16, can you run the original code? If not, then it might be that Keras has updated the API since this code was written. Permute() might no longer work the way the original code expects.

@likejazz
Copy link

@Ashima16
It needs another pair of brackets. try this.

model2.add(Permute((2, 1)))

@seanxiangct
Copy link

I found this code cannot work properly when the input of the LSTM is masked.
How can I solve it with masked input?

Hi @zzjin13,

I encountered the same issue with the masked input, but I don't think this is related to the implementation of the attention model.

As attention model is trying to learn the weightings of the inputs based on the input, a masked input with leading/trailing 0s in the sequence will trick the attention model to make it think this unique pattern is what is should pay attention to, resulting in your model is only paying attention to the mask not the actual inputs.

I would recommend applying sample weighting to your training sequence so the mask wouldn't contribute to the gradient update. You can pass a weighting matrix to the sample_weight argument of the model.fit() function

@kzhang123
Copy link

for i in range(a0):
for k in range(a2):
c[i][k]=a[i,:,k]*b[i]

So, three loops can be reduced to two.

@siddhartha-mukherjee-india

I wrote following code with the inspiration from the above discussion.

print('Build model...')
_input = Input(shape=[maxlen],)
embedded = Embedding(
input_dim=27,
output_dim=embedding_size,
input_length=maxlen,
trainable=True,
mask_zero=False
)(_input)

activations = LSTM(units, dropout=0.4, return_sequences=True)(embedded)
activations = Dropout(0.4)(activations)
attention = Dense(1, activation='tanh')(activations)
attention = Flatten()(attention)
attention = Activation('softmax')(attention)
attention = RepeatVector(units)(attention)
attention = Permute([2, 1])(attention)
``
#sent_representation = merge([input, attention], mode='mul')
`# Compilation error, so changed to Multiply()`
`sent_representation = Multiply()([activations, attention])`
`sent_representation = K.sum(sent_representation, axis=2)`
`# whatever value i put in axis, I get error. Please help.`

sent_representation = Dropout(0.4)(sent_representation)

prediction = Dense(numclasses, activation='softmax')(sent_representation)

model = Model(inputs=_input, outputs=prediction)
# Checkpoint to pick the best model.
checkpoint = ModelCheckpoint(filepath=CHECK_PT_FILE, verbose=1, monitor='val_loss',save_best_only=True, mode='auto')
model.compile(loss='categorical_crossentropy',
optimizer='adamax',
metrics=['accuracy'])

I need help to resolve the issue with K.Sum(). If I remove that, I am able to start the training.
But for attention, I belive that K.Sum() is needed.
Can any please help me to resolve this issue ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests