about embedding matrix structure

[tongjinle123]

Thanks for the brillient code.
I have noticed a santence in the paper:

"all the out-of-vocabulary words are mapped to a token ,whose embedding is trainable with random initialization." which not in your code. (they used a pretrained matrix)That seems make sence.
Do that works for the model?

[ghost]

Hi @tongjinle123 , you're right about fine tuning the unknown_word vectors. The paper suggests learning the unknown word vectors and we should do it.
I am just not too sure what the most elegant way to do this is in tensorflow. I am open to suggestions so if anyone has a brilliant idea to do this please let me know. Or even better, pull request!

Problem:
We want to fix word embeddings but fine-tune 1 word vector for token "Unk" (token for all unknown words). We only look up with 1 word embeddings vector which makes partial training difficult.

[tongjinle123]

To be frankly, i am wondering if keeping the pre-trained matrix freezed is reasonable.I think that no one could ensure the matrix is well trained for our task. Why not taking a second train?

Could you please share a paper,artical or experiment which is talking about this problem with me?
When it comes to the way to the implemention, i had a mind-storm-thought that define pre-trained matrix and unk matrix separately using tf.get_variable, run a concat op before trainop and run a assign op after trainop assigning the unk value to the unk matrix. I think that will not consume much time.

sorry for the bad comment format.... QAQ

[ghost]

Refer to #13 for this issue.