GloVe

[ynqa]

Overview

• Implementation GloVe (on memory)
  • Solver is prepared SGD and AdaGrad.

Feature Works

• GloVe with External Memory (write to disk once before training) like lexvec for huge data.

[chewxy]

left you some things to think about

[chewxy]

To think about: is it necessary to keep a copy of the tensors of the gradients?

[ynqa]

Yes. GloVe uses AdaGrad as the optimizer, instead of SGD. AdaGrad also use all past iteration's gradients to decrease learning rate automatically.
Here: https://en.wikipedia.org/wiki/Stochastic_gradient_descent#AdaGrad

[chewxy]

Yeah I'm familiar with Adagrad. I'm thinking maybe this shouldn't be put into the Embedding struct. Instead have something like a Solver struct

[ynqa]

ah I see, that's right. I'll try it.
(It's also good to select other optimizers, AdaDelta, Adam, and so on.)

[chewxy]

bad habit! should always check for errors.

Perhaps create a utility struct somewhere that looks like this:

type maybe struct{
    err error
}
func (m *maybe) DoBinOp(fn func(a, b interface{})(tensor.Tensor, error), a, b tensor.Tensor) (retVal tensor.Tensor) {
    if m.err != nil {
        return nil
    }
    retVal, m.err = fn(a, b)
    return
}

func (m *maybe) Error() error { return m.err }

then you can do this:

m := new(maybe)
inner := m.Do(tensor.Inner, g.emb.p[pind], g.emb.Q[qind])
bias := m.Do(tensor.Add, g.emb.biasP[pind, g.emb.biasQ[qind])
// and so on and so forth

[ynqa]

Sure.

[chewxy]

should be called Ones... it already returns a Tensor, so no need to repeat

[ynqa]

Sure.

[coveralls]

Coverage decreased (-10.0%) to 31.199% when pulling f6192c6 on glove into d310aec on master.

[coveralls]

Coverage increased (+1.6%) to 42.793% when pulling 844fcd1 on glove into d310aec on master.

[chewxy]

has there any work been done to compare this with a FMA function?

[ynqa]

Done a little experiments: against about 30 million records
with FMA in tensor: 1 min per iteration
without FMA (this): 30 sec per iteration

[chewxy]

To make this even faster check this out: https://blog.chewxy.com/2017/07/12/21-bits-english/. I'll help you convert when I find the time

[ynqa]

I'll watch it, thanks!

[chewxy]

While GloVe is the proper name (Global Vector) I think it's quite terrible to have random Uppercase letters in the middle of a name that is not a word. Perhaps stick with Glove ? Just an opinion

[ynqa]

Nope, so I'll rename it, by the way, why is it terrible there are uppercases in the middle?

[coveralls]

Coverage increased (+1.8%) to 43.019% when pulling 5f85057 on glove into d310aec on master.

[coveralls]

Coverage increased (+1.9%) to 43.062% when pulling 98ed1c7 on glove into d310aec on master.

[coveralls]

Coverage increased (+1.8%) to 43.029% when pulling 3982f37 on glove into d310aec on master.