Make lasagne.utils.create_param() get callable that returns Theano expressions or shared variables #695

erfannoury · 2016-06-09T12:53:00Z

This issues comments raised in #693

erfannoury · 2016-06-09T13:53:38Z

BTW, I have a question about naming in the PR that isn't yet resolved. What should I do with the name parameter if the callable method returns a shared variable or a Theano expression?

erfannoury · 2016-06-09T21:09:23Z

I have no idea why there are too many failures 😕

benanne · 2016-06-10T08:27:26Z

There seem to be a bunch of failures, and not just in tests for create_param, so that's odd. Have you run the tests locally?

Regarding the name parameter: our assumption is that, when a shared variable is passed in, it may or may not already have been named. If it already has a name, we do not want to overwrite it, so we do nothing. We could technically check if the specified variable has a name, and then insert a name if it doesn't have one, but that would get a bit hairy, I think. Also it wouldn't work for expressions.

erfannoury · 2016-06-10T10:19:52Z

I have only run the test locally for the parts of the code that I have changed. Maybe the failures are affected by the update to the create_param. I will re-check everything now.

Yes, it's written in the comments. My question was specifically for the case that a callable has been passed to the create_param which returns a Theano expression or a shared variables. In this case, should we ignore the name or set it? I think checking for the presence of name is a good solution.

erfannoury · 2016-06-10T13:13:29Z

@benanne One particular thing I encountered when running tests locally vs. Travis CI is that some tests won't pass when Theano is configured to use GPU. What should be done with these test? Some of the tests that previously were in the repository (not added by this PR) are prone to this problem, too.

erfannoury · 2016-06-11T12:04:45Z

@benanne I wonder how these two tests passed before:

lasagne/tests/layers/test_special.py::TestParametricRectifierLayer::test_alpha_init
lasagne/tests/layers/test_special.py::TestParametricRectifierLayer::test_get_output_for

In both tests, a scalar is passed to create_param, however I think (and I even ran the abovementioned tests using the original source code) these two tests should fail anyway. What am I missing here?

Now the build failures are down to these two and a doctest error on the EmbeddingLayer (lasagne/layers/embedding.py: lasagne.layers.embedding.EmbeddingLayer)

f0k · 2016-06-14T15:16:29Z

What am I missing here?

You changed the behaviour of create_param(). Before, when it got a callable, it called the callable and converted the result with lasagne.utils.floatX. Now you call the callable and run the result through the if cases, which don't grip for Python scalars. You'll have to keep your modifications to the original third if case or change the function to also accept Python scalars. Something like this:

if spec is callable:
    spec = spec(shape)
if spec is Python scalar:
    spec = floatX(spec)
if spec is numpy array:
    check for correct shape
    spec = theano.shared(spec)
if spec is theano variable:
    check for correct dimensionality
    add name if unset
else:
    complain that the spec is unsupported

Note that the if cases are not elif cases.

Instead of setting a boolean flag to spit out the correct error message in the end, you could gradually build up a prefix string for the error message inside the if cases ("the numpy array", "the Theano variable", "the numpy array returned by the callable", ...).

erfannoury · 2016-06-14T20:01:47Z

Ok, @f0k thanks for the suggestions. All tests pass now except the EmbeddingLayer's doctest, which I'm looking into now. I'll try updating the way error messages are handled, too.

erfannoury · 2016-06-14T20:17:29Z

I can't reproduce the doctest failure in EmbeddingLayer in either CPU or GPU mode on my computer.

f0k · 2016-06-15T10:51:13Z

lasagne/utils.py

+        spec = spec(shape)
+        callable_prefix = "cannot initialize parameters: the provided " \
+                          "callable did not return a value with the " \
+                          "correct shape or dimensions."


Just to make the errors a little nicer:

err_prefix = "cannot initialize parameter %s: " % name if callable(spec): spec = spec(shape) err_prefix += "the %s returned by the provided callable" else: err_prefix += "the provided %s" ... if isinstance(spec, np.ndarray): if spec.shape != shape: raise ValueError("%s has shape %s, should be %s") % (err_prefix % "numpy array", spec.shape, shape) ... if isinstance(spec, theano.Variable): if spec.ndim != len(shape): raise ValueError("%s has %d dimensions, should be %d" % (err_prefix % "Theano variable", spec.ndim, len(shape)) else: if "callable" in err_prefix: raise TypeError("%s is not a numpy array or a Theano expression" % (err_prefix % "value")) else: raise TypeError("%s is not a numpy array, a Theano expression, or a callable" % (err_prefix % "value"))

Note that I've changed RuntimeError to ValueError or TypeError (we need to clean this up elsewhere as well), the tests need to change accordingly.

erfannoury · 2016-06-15T14:06:10Z

Changed the error handling to what @f0k suggested. Tests are also updated accordingly and all pass. Only the EmbeddingLayer's doctest fails, however I can't reproduce it locally.

f0k · 2016-06-15T15:50:59Z

lasagne/utils.py

    """
    shape = tuple(shape)  # convert to tuple if needed
    if any(d <= 0 for d in shape):
        raise ValueError((
            "Cannot create param with a non-positive shape dimension. "
            "Tried to create param with shape=%r, name=%r") % (shape, name))

+    err_prefix = "cannot initialize parameter %s: " % name
+    if hasattr(spec, '__call__'):


I'd prefer if callable(spec).

Ok. I'll change this, too.

f0k · 2016-06-15T16:02:38Z

I can't reproduce the doctest failure in EmbeddingLayer in either CPU or GPU mode on my computer.

This is still a mystery. Does the new create_param change the dtype of a plain numpy array (i.e., is it passed through floatX)? Did you try running the test locally with THEANO_FLAGS=device=cpu,floatX=float64?

f0k · 2016-06-15T16:06:13Z

lasagne/utils.py

+        err_prefix += "the provided %s"
+
+    if np.isscalar(spec):
+        spec = floatX(spec)


np.isscalar("asdf") returns True. We need to explicitly check for numbers.

erfannoury · 2016-06-15T19:28:12Z

All tests are passing now. @f0k could you please review the code once more?

coveralls · 2016-06-15T19:36:12Z

Coverage remained the same at 100.0% when pulling cd65a2a on erfannoury:th-expr-param into 8fe645d on Lasagne:master.

f0k · 2016-06-16T16:46:37Z

lasagne/utils.py

+        err_prefix += "the provided %s"
+
+    if isinstance(spec, numbers.Number):
+        spec = np.asarray(spec)


Nice find! This seems slightly too strict, though (I'm sorry that this PR turns out to be much more intricate than it seemed at first glance):

>>> x = np.float32(123) >>> isinstance(x, numbers.Number) False

Maybe:

if isinstance(spec, numbers.Number) or isinstance(spec, np.generic) and spec.dtype.kind in 'biufc':

Although this is getting silly. Let's leave it at what you have.

(I'm sorry that this PR turns out to be much more intricate than it seemed at first glance)

No, it's ok. Sorry that I have been slow in implementing the correct code. This is my first PR for Lasagne 😄

>>> x = np.float32(123) >>> isinstance(spec, numbers.Number) False

@f0k I think this code isn't completely correct. numbers.Number in fact correctly handles the case you have provided.

>>> x = np.float32(123) >>> isinstance(x, numbers.Number) True

As far as I tested multiple cases, numbers.Number correctly handles scalars of multiple types and kinds.

I think this code isn't completely correct. numbers.Number in fact correctly handles the case you have provided.

Interesting. Not for me on Python 2.7.6 or 3.4.2, numpy 1.8.2.

You've passed spec as argument to isinstance, instead of x. I'm working on Python 2.7.11, NumPy 1.11.0.

No, I had just written it wrongly, I passed x. It depends on the numpy/Python version than. In any case, it indicates we don't need to do such a stretch here and can keep it as you have it.

But it won't work correctly in those combinations of NumPy/Python.
So I should keep them as is?

But it won't work correctly in those combinations of NumPy/Python.

It's a corner case. Using scalars was not actually documented, it just happened to be used in one of the tests. Using numpy scalars instead of Python scalars is even more undocumented.

So I should keep them as is?

You can also add my modified if statement, as you please. Either way is fine by me!

Yes, I think that's a more robust solution. I'll update the if statement.

coveralls · 2016-06-16T18:25:43Z

Coverage remained the same at 100.0% when pulling d50bd94 on erfannoury:th-expr-param into 8fe645d on Lasagne:master.

coveralls · 2016-06-16T19:40:26Z

Coverage remained the same at 100.0% when pulling 1955df9 on erfannoury:th-expr-param into 8fe645d on Lasagne:master.

coveralls · 2016-06-17T00:22:32Z

Coverage remained the same at 100.0% when pulling 169d6f7 on erfannoury:th-expr-param into 8fe645d on Lasagne:master.

f0k · 2016-06-17T08:41:32Z

lasagne/utils.py

        Either of the following:

-        * a numpy array with the initial parameter values
+        * an scalar or a numpy array with the initial parameter values


*a scalar (word starts with consonant)

f0k · 2016-06-17T08:57:40Z

Very cool! Just three minor requests. In the end, could you please squash this into a single commit? It should go like this:

git reset d5fbd64  # set HEAD to the first commit of this PR
git commit -a --amend -m 'Rewrite create_param() to allow callables returning Theano variables'  # amend all other changes to that commit
git push --force origin HEAD  # overwrite PR with the new clean history

coveralls · 2016-06-17T09:23:15Z

Coverage remained the same at 100.0% when pulling 776ce5f on erfannoury:th-expr-param into 1565826 on Lasagne:master.

erfannoury · 2016-06-17T09:24:44Z

@f0k squashed as you said, and all tests are passing.

f0k · 2016-06-17T10:02:23Z

Great, thanks a lot, everything looks fine! Will merge in the evening if there are no further comments.

f0k · 2016-06-17T17:11:36Z

Merged, thanks again!

twiecki · 2016-06-17T17:15:39Z

👍

erfannoury mentioned this pull request Jun 9, 2016

Bridge PyMC3 and Lasagne #693

Open

f0k reviewed Jun 15, 2016
View reviewed changes

f0k reviewed Jun 16, 2016
View reviewed changes

f0k reviewed Jun 17, 2016
View reviewed changes

Rewrite create_param() to allow callables returning Theano variables

776ce5f

f0k merged commit b2f42fd into Lasagne:master Jun 17, 2016

benanne mentioned this pull request Aug 8, 2016

Bayes by Backprop #731

Closed

Make lasagne.utils.create_param() get callable that returns Theano expressions or shared variables #695

Make lasagne.utils.create_param() get callable that returns Theano expressions or shared variables #695

Conversation

erfannoury commented Jun 9, 2016

erfannoury commented Jun 9, 2016

erfannoury commented Jun 9, 2016

benanne commented Jun 10, 2016

erfannoury commented Jun 10, 2016

erfannoury commented Jun 10, 2016

erfannoury commented Jun 11, 2016

f0k commented Jun 14, 2016

erfannoury commented Jun 14, 2016

erfannoury commented Jun 14, 2016

f0k Jun 15, 2016 • edited Loading

Choose a reason for hiding this comment

erfannoury commented Jun 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

f0k commented Jun 15, 2016

Choose a reason for hiding this comment

erfannoury commented Jun 15, 2016

coveralls commented Jun 15, 2016 • edited Loading

f0k Jun 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

f0k Jun 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jun 16, 2016 • edited Loading

coveralls commented Jun 16, 2016 • edited Loading

coveralls commented Jun 17, 2016 • edited Loading

f0k Jun 17, 2016 • edited Loading

Choose a reason for hiding this comment

f0k commented Jun 17, 2016

coveralls commented Jun 17, 2016 • edited Loading

erfannoury commented Jun 17, 2016

f0k commented Jun 17, 2016

f0k commented Jun 17, 2016

twiecki commented Jun 17, 2016

f0k Jun 15, 2016 •

edited

Loading

coveralls commented Jun 15, 2016 •

edited

Loading

f0k Jun 16, 2016 •

edited

Loading

f0k Jun 16, 2016 •

edited

Loading

coveralls commented Jun 16, 2016 •

edited

Loading

coveralls commented Jun 16, 2016 •

edited

Loading

coveralls commented Jun 17, 2016 •

edited

Loading

f0k Jun 17, 2016 •

edited

Loading

coveralls commented Jun 17, 2016 •

edited

Loading