[frames_from_files] ValueError: largest specified index is larger than trajectory length

[sonyahanson]

Getting ValueError: largest specified index is larger than trajectory length even with staged trajectories.

@maxentile and I had seen this for non-staged trajectories that we had reason to believe were being continuously updated. This was mentioned previously (#892). More recently this problem has been happening with trajectories that have been copied to a different folder and aren't actively updated.

traj = frames_from_files(trajfiles, top, indexes, chunksize, stride, reader=reader)
 File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/pyemma/coordinates/data/util/frames_from_file.py", line 111, in frames_from_files
   length))
ValueError: largest specified index (436 * stride=436 * 1=436) is larger than trajectory length '/cbio/jclab/home/hansons/sims/AZ/SYK/11407/run7-clone19.h5' = 240

Also, of note, if I try and run the same analysis on just that trajectory, the error does not occur and everything runs fine. There is also so far no evidence that the HDF5 file itself is bad.

Also note pyemma version:

conda list| grep pyemma
pyemma                    2.2.6               np110py27_0    omnia

[marscher]

The check in the function is correct, I've double checked it. So the question is why the indices you're passing in are out of bounds. Does it also happen if you use two trajectories (the "faulty" one + an arbitrary or subsequent one)?

@franknoe the test here only uses one trajectory, so maybe this case is not covered for hmm?
https://github.com/markovmodel/PyEMMA/blob/devel/pyemma/msm/tests/test_hmsm.py#L387-L398

[sonyahanson]

Hmmm. No error with two trajectories (the "faulty" one + an arbitrary or subsequent one).