indexed_gzip.indexed_gzip.ZranError: zran_seek returned error: -1 #801
Comments
|
@pauldmccarthy Any pointers for how to start debugging this? And any notion whether to treat this as a problem in nilearn, nibabel or indexed_gzip? |
|
Hey guys, I know nothing about For how long has the image been open before this error occurs, and what else has been done with/to it? What version of cd path/to/indexed_gzip
python setup.py build_ext -DZRAN_VERBOSE
python setup.py develop
export PYTHONPATH=`pwd` |
|
The dependency is I'm trying to generate a situation with @oesteban have you verified that the inputs to that |
|
This seems a problem only in lonestar5. I'll close for now and reopen if it happens again. |
|
Hey guys, The latest version of Thanks! |
oesteban
added a commit
to oesteban/fmriprep
that referenced
this issue
Nov 6, 2017
|
I got this error in one of the subjects with fmriprep 1.1.8 and indexed_gzip 0.8.7 ... Any suggestions? Thanks! |
|
@metad can you provide a stack trace, and ideally share the offending image? |
|
I just ran the same fmriprep command again with the same input files, but somehow got this error in a different workflow this time. Error info for the first time: The second time: I guess fmriprep was perhaps feeding some intermediate results to indexed_gzip, but there are quite a few intermediate files in the work folder and I'm not sure which one it's supposed to be... |
|
I'm guessing you're using nibabel 2.3.0 - it is worth upgrading to 2.3.1, as there were a couple of bugfixes related to how If that doesn't work, it would help if you could identify and share one of the offending files - it's probably best to share it with me by opening an issue over at pauldmccarthy/indexed_gzip. |
|
@metad your execution environment may also be relevant. For instance, I believe the issue we had before was on NFS shares. |
|
I updated nibabel to 2.3.1 and tried again, but still got that error. This time it also comes with a log file though, so maybe it provides more useful information (see below)? Would it be useful if I share the 406 input files? @effigies if it's related to the environment, would you expect I also get this error elsewhere? I've been only getting it with one subject (out of 25) every time I run it. I do have a NFS file system though. |
|
@metad sure, if you could share the offending files, I'll take a look at them. |
|
@pauldmccarthy The files are here (input_image in the last error log). Thank you!! |
|
There seems to be nothing wrong with any of the inputs - they all load fine on my laptop: import numpy as np
import indexed_gzip as igzip
import nibabel as nib
print(igzip.__version__)
print(nib.__version__)
for f in glob('*.gz'):
i = igzip.IndexedGzipFile(f)
d = nib.volumeutils.array_from_file((82, 82, 60), '<i2', i, 352)
print(f, d.shape, d.min(), d.max())Does the error always occur with the same subject? Can you try running from a local (non-networked) file system to see if it still occurs? |
|
Hmmm interesting... Yes it always happens with this one subject only. My local computer doesn't have enough memory, but I'll try to run it on OpenNeuro and see if that works. Thanks! |
|
I just realized OpenNeuro doesn't support running fmriprep on it any more. But somehow when I tried for a 4th time on my cluster, it didn't give any error.. Everything is same as before so I have no idea, but maybe we could close this issue for now... Thanks for the help. |
|
@metad OpenNeuro moved to a new storage backend (the old one was tied to the execution backend), so the new execution backend is in progress. There are plans for a limited, stop-gap execution strategy, but I don't know the timeline on that. Thanks for looking into this, and sorry there wasn't anything "fixable" this time. My suspicion is that there are some weird interactions with random access on NFS, probably cache-related. Hopefully next time we start running into it, we can get a cluster admin who can do some deep digging. It occurs to me just now that this may be an interaction of NFS caching and persistent nibabel filehandles. I'll open a new PR to remove an outdated setting. |
effigies
added a commit
that referenced
this issue
Oct 30, 2018
Keeping the file handle open within indexed_gzip may be producing the issue reported in gh-801. This will have us rely strictly on indexed_gzip's internal file handle management, and keep nibabel out of it.
|
@metad |
|
@pauldmccarthy I tried again and no error this time :) Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
DS000115 / sub-31
DS000119 / sub-74
DS000052 / sub-08
DS000121 / sub-04
Seems like a bad interaction between nilearn and the indexed_gzip or memory errors.
The text was updated successfully, but these errors were encountered: