Fix/remove confusing __iter__ implementation in theano.tensor.var.TensorVariable

[ghost]

I'm trying to translate Statistical Rethinking from R and RStan to Python and PyMC3.

On page 304, there's a simple logistic regression example. Data being used (publicly available):

	dept	applicant.gender	admit	reject	applications	is_male
1	A	male	512	313	825	1
2	A	female	89	19	108	0
3	B	male	353	207	560	1
4	B	female	17	8	25	0
5	C	male	120	205	325	1
6	C	female	202	391	593	0
7	D	male	138	279	417	1
8	D	female	131	244	375	0
9	E	male	53	138	191	1
10	E	female	94	299	393	0
11	F	male	22	351	373	1
12	F	female	24	317	341	0

Data is stored in a pandas DataFrame object. When I try and fit the model using:

with pm.Model() as m106:
    
    alpha = pm.Normal('alpha', 0, 10)
    beta_m = pm.Normal('beta_m', 0, 10)
    
    lin = alpha + beta_m * data['is_male']
    p = np.exp(lin) / (1 + np.exp(lin))
    
    admit = pm.Binomial('admit', n=data['applications'], p=p, observed=data['admit'])
    
    m106_map = pm.find_MAP()
    m106_traces = pm.sample(1000, start=m106_map)

I get the following error (which seems similar to pymc-devs/pymc#918):

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/type.py in dtype_specs(self)
    266                 'complex64': (complex, 'theano_complex64', 'NPY_COMPLEX64')
--> 267             }[self.dtype]
    268         except KeyError:

KeyError: 'object'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in constant_or_value(x, rtype, name, ndim, dtype)
    407             rval = rtype(
--> 408                 TensorType(dtype=x_.dtype, broadcastable=bcastable),
    409                 x_.copy(),

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/type.py in __init__(self, dtype, broadcastable, name, sparse_grad)
     49         self.broadcastable = tuple(bool(b) for b in broadcastable)
---> 50         self.dtype_specs()  # error checking is done there
     51         self.name = name

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/type.py in dtype_specs(self)
    269             raise TypeError("Unsupported dtype for %s: %s"
--> 270                             % (self.__class__.__name__, self.dtype))
    271 

TypeError: Unsupported dtype for TensorType: object

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in as_tensor_variable(x, name, ndim)
    201     try:
--> 202         return constant(x, name=name, ndim=ndim)
    203     except TypeError:

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in constant(x, name, ndim, dtype)
    421     ret = constant_or_value(x, rtype=TensorConstant, name=name, ndim=ndim,
--> 422                             dtype=dtype)
    423 

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in constant_or_value(x, rtype, name, ndim, dtype)
    416     except Exception:
--> 417         raise TypeError("Could not convert %s to TensorType" % x, type(x))
    418 

TypeError: ('Could not convert 1     Elemwise{mul,no_inplace}.0\n2     Elemwise{mul,no_inplace}.0\n3     Elemwise{mul,no_inplace}.0\n4     Elemwise{mul,no_inplace}.0\n5     Elemwise{mul,no_inplace}.0\n6     Elemwise{mul,no_inplace}.0\n7     Elemwise{mul,no_inplace}.0\n8     Elemwise{mul,no_inplace}.0\n9     Elemwise{mul,no_inplace}.0\n10    Elemwise{mul,no_inplace}.0\n11    Elemwise{mul,no_inplace}.0\n12    Elemwise{mul,no_inplace}.0\nName: applications, dtype: object to TensorType', <class 'pandas.core.series.Series'>)

During handling of the above exception, another exception occurred:

AsTensorError                             Traceback (most recent call last)
<ipython-input-144-fb615dfa2e93> in <module>()
      7     p = np.exp(lin) / (1 + np.exp(lin))
      8 
----> 9     admit = pm.Binomial('admit', n=data['applications'], p=p, observed=data['admit'])
     10 
     11     m106_map = pm.find_MAP()

/Users/horatiu/anaconda/lib/python3.5/site-packages/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
     24         if isinstance(name, string_types):
     25             data = kwargs.pop('observed', None)
---> 26             dist = cls.dist(*args, **kwargs)
     27             return model.Var(name, dist, data)
     28         elif name is None:

/Users/horatiu/anaconda/lib/python3.5/site-packages/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
     37     def dist(cls, *args, **kwargs):
     38         dist = object.__new__(cls)
---> 39         dist.__init__(*args, **kwargs)
     40         return dist
     41 

/Users/horatiu/anaconda/lib/python3.5/site-packages/pymc3/distributions/discrete.py in __init__(self, n, p, *args, **kwargs)
     43         self.n = n
     44         self.p = p
---> 45         self.mode = tt.cast(tt.round(n * p), self.dtype)
     46 
     47     def random(self, point=None, size=None, repeat=None):

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in round(a, mode)
   2052     """round_mode(a) with mode in [half_away_from_zero, half_to_even]"""
   2053     if mode == "half_away_from_zero":
-> 2054         return round_half_away_from_zero(a)
   2055     elif mode == "half_to_even":
   2056         return round_half_to_even(a)

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
    609         """
    610         return_list = kwargs.pop('return_list', False)
--> 611         node = self.make_node(*inputs, **kwargs)
    612 
    613         if config.compute_test_value != 'off':

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/elemwise.py in make_node(self, *inputs)
    541         using DimShuffle.
    542         """
--> 543         inputs = list(map(as_tensor_variable, inputs))
    544         shadow = self.scalar_op.make_node(
    545             *[get_scalar_type(dtype=i.type.dtype).make_variable()

/Users/horatiu/anaconda/lib/python3.5/site-packages/theano/tensor/basic.py in as_tensor_variable(x, name, ndim)
    206         except Exception:
    207             str_x = repr(x)
--> 208         raise AsTensorError("Cannot convert %s to TensorType" % str_x, type(x))
    209 
    210 # this has a different name, because _as_tensor_variable is the

AsTensorError: ('Cannot convert 1     Elemwise{mul,no_inplace}.0\n2     Elemwise{mul,no_inplace}.0\n3     Elemwise{mul,no_inplace}.0\n4     Elemwise{mul,no_inplace}.0\n5     Elemwise{mul,no_inplace}.0\n6     Elemwise{mul,no_inplace}.0\n7     Elemwise{mul,no_inplace}.0\n8     Elemwise{mul,no_inplace}.0\n9     Elemwise{mul,no_inplace}.0\n10    Elemwise{mul,no_inplace}.0\n11    Elemwise{mul,no_inplace}.0\n12    Elemwise{mul,no_inplace}.0\nName: applications, dtype: object to TensorType', <class 'pandas.core.series.Series'>)

The error is not very informative, and doesn't seem to point to my code.
But when using:

with pm.Model() as m106:
    
    alpha = pm.Normal('alpha', 0, 10)
    beta_m = pm.Normal('beta_m', 0, 10)
    
    lin = alpha + beta_m * data['is_male']
    p = np.exp(lin) / (1 + np.exp(lin))
    
    admit = pm.Binomial('admit', n=data['applications'].values, p=p, observed=data['admit'])
    
    m106_map = pm.find_MAP()
    m106_traces = pm.sample(1000, start=m106_map)

So explicitly passing in the numpy array rather than the pandas Series:

admit = pm.Binomial('admit', n=data['applications'].values, p=p, observed=data['admit'])

Everything works as expected. I'm just trying to figure out why that is? Is there a reference in the documentation for this behavior? What is the lesson I should take away from this? :)

Using pandas 0.19, numpy 1.11, pymc3.0rc2

[twiecki]

Hm, what does data['applications'].dtype say? Seems like it somehow casts to the wrong type. If it's not that it seems to be some incompabitility with theano.

In any case, this is certainly not expected and an odd error. Unfortunately the pymc3 veneer on top of theano is thin so when something breaks you are left with theano's error messages which can be hard to read.

[ghost]

data['applications'].dtype is dtype('int64').

[springcoil]

Hi @Horatiu could you try converting that to int32 and see if that works?

On Tue, Nov 1, 2016 at 1:28 PM, Horatiu notifications@github.com wrote:

data['applications'].dtype is dtype('int64').

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1492#issuecomment-257554419,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8DiMsT5dJHdUGmFb4LSTu4Qyt3Vf00ks5q5zBigaJpZM4KmANQ
.

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

[ghost]

Hi @springcoil, converted and same issue.

[ghost]

Please consider the issue (very) low priority -- I have a working example, I just want to understand why passing the numpy array explicitly is required.

[twiecki]

Interestingly, it works if you pass it to observed. I think we have some logic in place there as it's the more common use case.

[springcoil]

Is this something that should go into the pymc-devs/pymc#1488 example?

[aloctavodia]

Closing as I am not able to reproduce this error (feel free to reopen is still a problem). @RogerTheAlien you may want to check this repository.

[tolex3]

Hi, I know I'm coming late here, but I see the same problem: in some cases, pymc3 fails to use pandas series, throwing a value error:

ValueError: length not known: Elemwise{mul,no_inplace} [id A] ''   
 |TensorConstant{[ 1.325014...71205771]} [id B]
 |InplaceDimShuffle{x} [id C] ''   
   |beta [id D]

# cell 1:
but when taking values of the series, that is using df['x'].values, it works. Here's code to demo it:
import numpy as np
import pandas as pd
import pymc3 as pm

%load_ext watermark
%watermark -n -u -v -iv -w

# output cell 1: 
pymc3  3.9.3
numpy  1.19.1
pandas 1.1.0
last updated: Sat Aug 15 2020 

CPython 3.8.5
IPython 7.17.0
watermark 2.0.2

#cell 2:
df = pd.DataFrame({'x' : np.random.normal(0,1,size=100),
                  'y' : np.random.normal(0,1,size=100)})

#cell 3:
model = pm.Model()

with model:
    
    alpha = pm.Normal('alpha',mu=0,sd=1)
    beta = pm.Normal('beta',mu=0,sd=1)
    sigma = pm.Uniform('sigma',0,1)
    
    reg = alpha + df['x'] * beta
    
    lkh = pm.Normal('lkh',mu=reg,sd=sigma,observed=df['y'])
    
    trace = pm.sample(500,tune=100)

#cell 3 output:
ValueError                                Traceback (most recent call last)
<ipython-input-11-eb5f40f5e35b> in <module>
      7     sigma = pm.Uniform('sigma',0,1)
      8 
----> 9     reg = alpha + df['x'] * beta
     10 
     11     lkh = pm.Normal('lkh',mu=reg,sd=sigma,observed=df['y'])

/usr/local/lib64/python3.8/site-packages/pandas/core/ops/common.py in new_method(self, other)
     63         other = item_from_zerodim(other)
     64 
---> 65         return method(self, other)
     66 
     67     return new_method

/usr/local/lib64/python3.8/site-packages/pandas/core/ops/__init__.py in wrapper(left, right)
    343         result = arithmetic_op(lvalues, rvalues, op)
    344 
--> 345         return left._construct_result(result, name=res_name)
    346 
    347     wrapper.__name__ = op_name
/usr/local/lib64/python3.8/site-packages/pandas/core/series.py in _construct_result(self, result, name)
   2755         # We do not pass dtype to ensure that the Series constructor
   2756         #  does inference in the case where `result` has object-dtype.
-> 2757         out = self._constructor(result, index=self.index)
   2758         out = out.__finalize__(self)
   2759 

/usr/local/lib64/python3.8/site-packages/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
    299                 raise TypeError(f"'{type(data).__name__}' type is unordered")
    300             else:
--> 301                 data = com.maybe_iterable_to_list(data)
    302 
    303             if index is None:

/usr/local/lib64/python3.8/site-packages/pandas/core/common.py in maybe_iterable_to_list(obj)
    277     """
    278     if isinstance(obj, abc.Iterable) and not isinstance(obj, abc.Sized):
--> 279         return list(obj)
    280     return obj
    281 

/usr/local/lib/python3.8/site-packages/theano/tensor/var.py in __iter__(self)
    638     def __iter__(self):
    639         try:
--> 640             for i in xrange(theano.tensor.basic.get_vector_length(self)):
    641                 yield self[i]
    642         except TypeError:
/usr/local/lib/python3.8/site-packages/theano/tensor/basic.py in get_vector_length(v)
   4826     else:
   4827         msg = str(v)
-> 4828     raise ValueError("length not known: %s" % msg)
   4829 
   4830 

ValueError: length not known: Elemwise{mul,no_inplace} [id A] ''   
 |TensorConstant{[ 1.325014...71205771]} [id B]
 |InplaceDimShuffle{x} [id C] ''   
   |beta [id D]

[twiecki]

Thanks, probably a theano issue. @brandonwillard any idea?

[tolex3]

Totally unrelated, but I just have to give the pymc-team a whole bunch of cudos: Ive been using pymc2 for the past 4-5 years, unable to upgrade due to my very weird setup - windows + cygwin - but decided to bite the bullet and migrate to Linux... installing pymc3 worked like a charm, and BOY am I impressed by the way pymc has evolved, lots of new functionality and supporting features for analysis, plotting etc... Im very impressed! —Tommy + sent from my iPhone

…

On Aug 17, 2020, at 10:06, Thomas Wiecki ***@***.***> wrote:  Thanks, probably a theano issue. @brandonwillard any idea? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[brandonwillard]

The problem appears to be the product df['x'] * beta. Convert df['x'] to a Theano object first.

[twiecki]

@brandonwillard But it would work if df['x'] were a numpy array, right? Wouldn't it be nice if Theano supported pandas Series and DataFrames in the same place it supports numpy arrays?

[brandonwillard]

I suppose we could consider adding that into our Theano fork, but only if it doesn't involve adding a new Pandas dependency.

[twiecki]

It should be easy to test whether x.values is a numpy array.

[brandonwillard]

Yeah, that's exactly what I was thinking. We could try adding this to the __r*__ methods on the tensor types (and the others, if they don't already attempt conversion).

[brandonwillard]

After some investigation, I've found that there is already support for the conversion of Pandas objects; however, the exact problem observed in this issue prevents use of the relevant logic.

Specifically, the class _tensor_py_operators—from which TensorVariables inherit—automatically provides an __iter__ method and this tricks Pandas (or any other library) into thinking that all TensorVariables are iterable (e.g. via isinstance(..., Iterable) and the like) when not all of them really are.

If you look at the implementation of __iter__, you'll see that it's ready to raise an exception when the TensorVariable isn't a vector (see theano.tensor.basic.get_vector_length). This is a misuse of typing, because it's advertising an interface that it doesn't necessarily provide, and determining when it can provide that interface isn't even all that difficult (e.g. at the very least, it requires TensorVariable.type.ndim == 1).

In other words, this design needs to be fixed. One simple approach is to create a VectorVariable subclass of TensorVariable with the __iter__ interface, then a factory could be used to instantiate those when type.ndim == 1. I imagine that most of the work behind that change would involve finding and fixing any overly strict type checks (e.g. type(x) == TensorVariable).

The aforementioned fix has been split off into its own issue: #93. That's the correct first step toward fixing this particular issue.

[brandonwillard]

Duplicate of #93