Scan cannot include MRG_RandomStreams.binomial with symbolic shape

[f0k]

scan cannot have a MRG_RandomStreams.binomial in its step function that uses the symbolic shape of a recurrent variable: It fails to compile with a MissingInputError. This was discovered by a Lasagne user some days ago (https://groups.google.com/forum/#!topic/lasagne-users/8E1iR04_nK8), and a few minutes ago a colleague stumbled upon the same problem, so I think it might deserve some attention. (I've read that dropouts in recurrent connections don't work well in practice, but this shouldn't be a reason to refuse compilation ;).)

The following code demonstrates that scan works fine with a nonsymbolic shape in binomial, but fails with a symbolic shape:

import theano
import theano.tensor as T
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
floatX = theano.config.floatX

inputs = T.matrix()
hid_init = T.vector()
rng = RandomStreams()

# Works:
def step(inp, prev):
    # rng.binomial with constant shape
    return inp + prev * rng.binomial((10,), p=.5, dtype=floatX)
out = theano.scan(step, sequences=[inputs], outputs_info=[hid_init])[0]
theano.function([inputs, hid_init], out)
print "Success!"

# Fails:
def step(inp, prev):
    # rng.binomial with symbolic shape
    return inp + prev * rng.binomial(prev.shape, p=.5, dtype=floatX)
out = theano.scan(step, sequences=[inputs], outputs_info=[hid_init])[0]
theano.function([inputs, hid_init], out)
print "Will never be reached!"

The error is:

theano.gof.fg.MissingInputError: ("An input of the graph, used to compute Shape(<TensorType(float32, vector)>), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.", <TensorType(float32, vector)>)

[carriepl]

The error comes from the fact that the 'updates' output of Scan is not provided to function(). It works with the following code :

# Works:
def step(inp, prev):
    # rng.binomial with constant shape
    return inp + prev * rng.binomial((10,), p=.5, dtype=floatX)
out, updates = theano.scan(step, sequences=[inputs], outputs_info=[hid_init])
theano.function([inputs, hid_init], out, updates=updates)
print "Success!"

# Fails:
def step(inp, prev):
    # rng.binomial with symbolic shape
    return inp + prev * rng.binomial(inp.shape, p=.5, dtype=floatX)
out, updates = theano.scan(step, sequences=[inputs], outputs_info=[hid_init])
theano.function([inputs, hid_init], out, updates=updates)
print "Will never be reached!"

[f0k]

Oh, of course. Interesting behaviour, though. Thanks a lot!

[carriepl]

I agree 100% that this is weird behaviour. We should either :
-find a way such that the user doesn't need to perform this updates management or
-find a way to detect the case where the user should have provided the updates and didn't and display an informative error message.

[f0k]

find a way such that the user doesn't need to perform this updates management

This would be great. For uses of RNGs outside of scan, updates are handled transparently already. If this isn't possible for scan, the API in Lasagne will have to become a little more complex to cater for that (not a deal breaker, but would make recurrent layers a bit cumbersome).

[apeterswu]

Hi,
I met the same error. So is there any solution if I need to use in scan?

[f0k]

So is there any solution if I need to use in scan?

As mentioned in #3437 (comment), it should be enough to capture the updates returned by theano.scan() and make sure they are passed on to theano.function() and not just discarded.