Loss function is not squared in glove_cython?

[ducovrossem]

Not sure if I am missing something here but thought I'd ask for clarification - the loss function is not squared.

loss = entry_weight * (prediction - c_log(count))

Also this implementation does not generate seperate vectors for when word is used in context?

[maciejkula]

Bad variable name. I think this is the gradient of the loss function.

Yes, this implementation does not generate separate vectors for context words. This makes it more memory efficient, as I can use an upper triangular matrix for the co-occurrence matrix (and only one matrix of vectors).

[ducovrossem]

Clear - thanks maciejkula.

[piskvorky]

FYI @maciejkula :
@dselivanov reports that this particular optimization (ignoring the context vectors) leads to a massive loss of accuracy:
http://rare-technologies.com/making-sense-of-word2vec/#comment-488

[maciejkula]

Interesting, I'll definitely have a look.

Incidentally, I think my more recent project (https://github.com/lyst/lightfm) should work really well on word embeddings (it uses a fancy learning-to-rank approach), I need to try it out.