ipynb is down

[cosmozhang]

The ipynb was not fixed..

[JonathanRaiman]

Thanks! Good catch. Just fixed a typo in the ipynb.

[cosmozhang]

Hi Jonathan,

 I am trying to implement dropout in the theano.scan, but without 

using your package, since I have written most part of the rnn. Do you
have any suggestions? I read your codes and instructions, but still have
some questions. E.g, since masks is non_sequences, what will be the
input in the theano.function?
Thanks very much!

Cosmo

On 04/20/2015 10:18 PM, Jonathan Raiman wrote:

Thanks! Good catch. Just fixed a typo in the ipynb.

—
Reply to this email directly or view it on GitHub
#11 (comment).

Best,
Cosmo Zhang

[JonathanRaiman]

Hey Cosmo,

To get dropout to work I've done the following:

Get some random number generator:

import theano, theano.tensor as T
srng = theano.tensor.shared_randomstreams.RandomStreams(1234)

Then create the mask you want to use:

drop_prob = T.scalar()
shape = (300, 300)
mask = T.cast(
    srng.binomial(n=1, p=1-drop_prob, size=shape),
    theano.config.floatX
)

And then in your scan:

result, updates = theano.scan(fn = step,
                     sequences = [x],
                     outputs_info = etc...,
                     non_sequences = [mask])

error = etc...

Your theano function can now take as inputs:

func = theano.function([x, target, drop_prob], (result[-1] - target)**2)

As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.

[cosmozhang]

Thank you very much, I solved it last night in another way:

import theano, theano.tensoras T
srng= theano.tensor.shared_randomstreams.RandomStreams(1234)

drop_prob= T.scalar()
shape= (shape1, 300,300) #shape1 is the length of input sequence
mask= T.cast(
srng.binomial(n=1,p=1-drop_prob,size=shape),
theano.config.floatX
)

and then

|result, updates = theano.scan(fn = step,
sequences = [x, mask],
outputs_info = etc...,
non_sequences = None)|

Do you think it is also plausible?

And in your setting, how did yo write step function?

Thank you very much!

On 4/21/2015 10:47 PM, Jonathan Raiman wrote:

Hey Cosmo,

To get dropout to work I've done the following:

Get some random number generator:

import theano, theano.tensoras T
srng= theano.tensor.shared_randomstreams.RandomStreams(1234)

Then create the mask you want to use:

drop_prob= T.scalar()
shape= (300,300)
mask= T.cast(
srng.binomial(n=1,p=1-drop_prob,size=shape),
theano.config.floatX
)

And then in your scan:

|result, updates = theano.scan(fn = step,
sequences = [x],
outputs_info = etc...,
non_sequences = [mask])

error = etc...
|

Your theano function can now take as inputs:

|func = theano.function([x, target, drop_prob], (result[-1] - target)**2)
|

As long as the random variables aren't generated within the scan op
you should be fine. I've had problems whenever I tried created a new
set of binomials on every time step or something else that's fancy.
But the frozen dropout values as shown above worked for me.

—
Reply to this email directly or view it on GitHub
#11 (comment).