Failing to read XDR offsets is a fatal error

[richardjgowers]

Sometimes loading xdr offsets fails.

  File "/home/user/miniconda2/envs/env/lib/python2.7/site-packages/MDAnalysis/coordinates/XDR.py", line 84, in read_numpy_offsets
    return {k: v for k, v in six.iteritems(np.load(filename))}
  File "/home/user/miniconda2/envs/env/lib/python2.7/site-packages/numpy/lib/npyio.py", line 429, in load
    "Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file '/host/user/dir/files/.file.xtc_offsets.npz' as a pickle

This shouldn't cause the entire file loading process to fail. It should be possible wrap offset loading in a try/except to regenerate the offsets if needed

[orbeckst]

If we put in an try/except then we should also spit out a error warning message with as much detail about the file as we can so that we might have a chance to figure out why this is happening.

[lilyminium]

For now, does deleting the offset file fix this by forcing offsets to regenerate?

[orbeckst]

I am saying yes (until someone proves me wrong) – this should always work.

[zemanj]

Better workaround: IIRC XDRBaseReader has a refresh_offsets kwarg which forces the reader to regenerate frame offsets.

[orbeckst]

You're correct @zemanj , MDAnalysis.coordinates.XDR.XDRBaseReader and therefore the XTCReader and TRRReader have refresh_offsets so one can just do

u = mda.Universe(data.TPR, data.XTC, refresh_offsets=True)

as demonstrated here by monitoring the last-modified time of the offset file:

>>> import MDAnalysis as mda; from MDAnalysis.tests import datafiles as data
>>> import os
>>> u = mda.Universe(data.TPR, data.XTC)
>>> offset_file = mda.coordinates.XDR.offsets_filename(u.trajectory.filename)
>>> print(offset_file)
~/anaconda3/envs/mda3/lib/python3.6/site-packages/MDAnalysisTests/data/.adk_oplsaa.xtc_offsets.npz
>>> print(os.path.getmtime(offset_file))
1573257681.0
>>> # load again: nothing changes
>>> u = mda.Universe(data.TPR, data.XTC)
>>> print(os.path.getmtime(offset_file))
1573257681.0
>>> # refresh: offset file is modified
>>> u = mda.Universe(data.TPR, data.XTC, refresh_offsets=True)
>>> print(os.path.getmtime(offset_file))
1573257862.0