pyemma.coordinates.load suddenly broke

[rafwiewiora]

Hi guys!

Not sure what's going on quite yet, but I've been seeing this since last night on pyemma-dev and I think pyemma conda packages (let me just do a complete clean up of the conda environment in a sec and confirm the release version is also doing this).

The error:

In [2]: x = pyemma.coordinates.load('run0-clone0.h5', top='top.pdb')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-6ccc7bc81304> in <module>()
----> 1 x = pyemma.coordinates.load('run0-clone0.h5', top='top.pdb')

/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/api.pyc in load(trajfiles, features, top, stride, chunk_size, **kw)
    201                  or len(trajfiles) is 0)):
    202         reader = create_file_reader(trajfiles, top, features, chunk_size=chunk_size, **kw)
--> 203         trajs = reader.get_output(stride=stride)
    204         if len(trajs) == 1:
    205             return trajs[0]

/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/iterable.pyc in get_output(self, dimensions, stride, skip, chunk)
    149                                     description='getting output of %s' % self.__class__.__name__,
    150                                     stage=1)
--> 151             for itraj, chunk in it:
    152                 L = len(chunk)
    153                 if L > 0:

/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/datasource.pyc in next(self)
    621 
    622     def next(self):
--> 623         X = self._it_next()
    624         while X is not None and (
    625                 (not self.return_traj_index and len(X) == 0) or (self.return_traj_index and len(X[1]) == 0)

/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/datasource.pyc in _it_next(self)
    601         self.state.pos = self.state.pos_adv
    602         try:
--> 603             X = self._use_cols(self._next_chunk())
    604         except StopIteration:
    605             self._last_chunk_in_traj = True

/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/feature_reader.pyc in _next_chunk(self)
    315         """
    316         chunk = next(self._mditer)
--> 317         shape = chunk.xyz.shape
    318 
    319         self._t += shape[0]

AttributeError: 'NoneType' object has no attribute 'xyz'

@jchodera also reporting the same thing with pyemma-dev:

[chodera@mskcc-ln1 ~]$ python testpyemma.py 
downloading alanine dipeptide from https://ndownloader.figshare.com/articles/1026131/versions/8 to /cbio/jclab/home/chodera/msmbuilder_data
Obtaining file info: 100% (10/10) [#########################################################################################################################################################################################################################################################################################################################] eta 00:01 -Traceback (most recent call last):
  File "testpyemma.py", line 18, in <module>
    xyz = pyemma.coordinates.load(fnames,top='ala_top.pdb')
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/api.py", line 203, in load
    trajs = reader.get_output(stride=stride)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/iterable.py", line 151, in get_output
    for itraj, chunk in it:
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/datasource.py", line 623, in next
    X = self._it_next()
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/datasource.py", line 603, in _it_next
    X = self._use_cols(self._next_chunk())
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/feature_reader.py", line 317, in _next_chunk
    shape = chunk.xyz.shape
AttributeError: 'NoneType' object has no attribute 'xyz'

while it's working ok for @maxentile with the pyemma package.

Any idea what's happening?

Thanks!

[franknoe]

Thanks for reporting. I don't know why the behavior might have changed
suddenly, as I don't see recent commits with changes in this code.
Latest change in feature_reader was 16 days ago.

However, it looks like this code branch runs into the issue of not
properly catching when an iterator runs empty (trying to get shape
information from a trajectory object that pops out of an iterator, but
is in fact None). @marscher or @clonker : any ideas?. @ rafwiewiora
https://github.com/rafwiewiora do you encounter this issue only with
this dataset or only h5 format, or more generally?

In any case, we should make sure that test cover a wider range of
inputs. Clearly, load is being tested, and the latest devel version
tested OK on Travis and AppVeyor:
https://travis-ci.org/markovmodel/PyEMMA
https://ci.appveyor.com/project/marscher/pyemma
However, I can't find a test of load that would cover anything
except for xtc, but maybe I'm not looking in the right place. I think we
decided to test save/load cycles across all mdtraj formats. @marscher or
@clonker , can you point me to that, or if we don't have it, add such a
test?

Am 30/04/16 um 01:37 schrieb Rafal Wiewiora:

Hi guys!

Not quite sure what's going on quite yet, but I've been seeing this
since last night on |pyemma-dev| and I think |pyemma| conda packages
(let me just do a complete clean up of the conda environment in a sec
and confirm the release version is also doing this).

The error:

In [2]: x = pyemma.coordinates.load('run0-clone0.h5', top='top.pdb')

AttributeError Traceback (most recent call last)
in () ----> 1 x =
pyemma.coordinates.load('run0-clone0.h5', top='top.pdb')
/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/api.pyc
in load(trajfiles, features, top, stride, chunk_size, *_kw) 201 or
len(trajfiles) is 0)): 202 reader = create_file_reader(trajfiles, top,
features, chunk_size=chunk_size, *_kw) --> 203 trajs =
reader.get_output(stride=stride) 204 if len(trajs) == 1: 205 return
trajs[0]
/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/iterable.pyc
in get_output(self, dimensions, stride, skip, chunk) 149
description='getting output of %s' % self.class.name, 150
stage=1) --> 151 for itraj, chunk in it: 152 L = len(chunk) 153 if L >
0:
/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/datasource.pyc
in next(self) 621 622 def next(self): --> 623 X = self._it_next() 624
while X is not None and ( 625 (not self.return_traj_index and len(X)
== 0) or (self.return_traj_index and len(X[1]) == 0)
/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/_base/datasource.pyc
in _it_next(self) 601 self.state.pos = self.state.pos_adv 602 try: -->
603 X = self._use_cols(self._next_chunk()) 604 except StopIteration:
605 self._last_chunk_in_traj = True
/Users/rafalpwiewiora/anaconda/lib/python2.7/site-packages/pyEMMA-2.1.1-py2.7-macosx-10.5-x86_64.egg/pyemma/coordinates/data/feature_reader.pyc
in _next_chunk(self) 315 """ 316 chunk = next(self._mditer) --> 317
shape = chunk.xyz.shape 318 319 self._t += shape[0] AttributeError:
'NoneType' object has no attribute 'xyz' |

@jchodera https://github.com/jchodera also reporting the same thing
with |pyemma-dev|:

|[chodera@mskcc-ln1 ~]$ python testpyemma.py downloading alanine
dipeptide from
https://ndownloader.figshare.com/articles/1026131/versions/8 to
/cbio/jclab/home/chodera/msmbuilder_data Obtaining file info: 100%
(10/10)
[#########################################################################################################################################################################################################################################################################################################################]
eta 00:01 -Traceback (most recent call last): File "testpyemma.py",
line 18, in xyz =
pyemma.coordinates.load(fnames,top='ala_top.pdb') File
"/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/api.py",
line 203, in load trajs = reader.get_output(stride=stride) File
"/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/iterable.py",
line 151, in get_output for itraj, chunk in it: File
"/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/datasource.py",
line 623, in next X = self._it_next() File
"/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/_base/datasource.py",
line 603, in _it_next X = self._use_cols(self._next_chunk()) File
"/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/pyEMMA-2.1.1+1.ged32b88-py2.7-linux-x86_64.egg/pyemma/coordinates/data/feature_reader.py",
line 317, in _next_chunk shape = chunk.xyz.shape AttributeError:
'NoneType' object has no attribute 'xyz' |

while it's working ok for @maxentile https://github.com/maxentile
with the |pyemma| package.

Any idea what's happening?

Thanks!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#790

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[franknoe]

@clonker has addressed this issue in #791. @rafwiewiora , please try again, and close this issue yourself if it works. Thanks again.

[rafwiewiora]

Thanks for such fast fix! Confirming pyemma-dev working ok now, but not the release package yet.

Closing this, thanks again!

[franknoe]

OK, we were planning a batch release anyway. Probably Mon/Tue.

Am 30/04/16 um 20:08 schrieb Rafal Wiewiora:

Thanks for such fast fix! Confirming |pyemma-dev| working ok now, but
not the release package yet.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#790 (comment)

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany