Implement NCE loss to speed up Softmax with large amount of labels

[xlvector]

For word based LSTM, the Softmax layer receive large amount of labels which make training very slow. Two methods can solve this problem: hierarchical softmax (HS) and NCE.

I think NCE is better than HS, so I want to implement NCE Loss similar to this repos: https://github.com/yandex/faster-rnnlm

The API may looks like

NCEOutput(data = input, label = target, noise = [negative samples])

Do you have more suggestions on this?

[sxjscience]

Tensorflow also has a nce_loss that we can refer to. https://www.tensorflow.org/versions/r0.9/api_docs/python/nn.html#nce_loss

[xlvector]

Thanks. Another problem I find is LSTM on words always cause OOM. However, NCE
should not affect memory.

How to I measure memory usage in SoftmaxOutput of LSTM with num_hidden, default_bucket_key and num_label. I think it should be num_hidden * num_label. But I find it seems LSTM use num_hidden * num_label * default_bucket_key

[piiswrong]

Is default_bucket_key = length? If you have output at each time step that would make sense

[xlvector]

yes. so if seq length is too long. it will cause oom？
i have vocab size 5000，hidden = 100，length=3000

[xlvector]

I think the NCE Loss operator should not replace Softmax, it should replace last FullConnect because we can not calculate nhidden * nlabel matrix in last FC.

So, NCE is FC + Softmax Loss, merge two op into single op. What's your ideas?

[xlvector]

I try to write an C++ NceLoss layer to do this, but find its too slow.
However, after refer to tensorflow's implement, I find we can use existing op to do NCE loss.

So, I will submit an python example for this.

[sxjscience]

That's good. I'm planning to create another folder to include some OPs that can be constructed by basic c++ ops (Like LSTM). NCE can be one of them.

[sxjscience]

Closed by #2756