LSTM Learnable Hidden State

[myaldiz]

Learnable hidden state feature, I did not thoroughly test whether accuracy for sequence_tagger gets better or not.

@alanakbik not sure how to initialize the hidden state though, just used rand for now

(Forgive me if I skipped some steps about the rules of PR, appreciate your guidance)

[alanakbik]

I've fixed some small issues with the code, so the unit tests now run through. I've also done a quick experiment using learnable hidden states but I cannot say if this makes much of a difference.

[myaldiz]

I've fixed some small issues with the code, so the unit tests now run through. I've also done a quick experiment using learnable hidden states but I cannot say if this makes much of a difference.

I think it can be a result of one of three reasons.

First one is if you tried with Conll-03, accuracy is already too high and default network is shallow, maybe benefits become unobservable. How about trying harder tasks with larger datasets using networks with larger capabilities?

Secondly, our initialization might be a problem. It can get better with some experimentation.

Thirdly, maybe it actually just does not make much difference :)

[alanakbik]

That's true :) Would you like to experiment with initialization, or should we go ahead and merge this as it is?

[myaldiz]

That's true :) Would you like to experiment with initialization, or should we go ahead and merge this as it is?

Let's just merge it as it is, and maybe change default parameter to false for not causing problems. And later if I can find some time, I will try to experiment and put the results here. At the same time, we can comment on the code so that maybe other people put their results about it.

[alanakbik]

👍

[yosipk]

👍