ValueError: cannot reshape array of size 0 into shape () when using system-wide scipy.netcdf #579
Comments
|
Ouch - minimal reproducible example: Would you consider putting in an issue to scipy ? I think this is a clash with netcdf and some recent changes to numpy. |
|
@matthew-brett submitted, see linked issue. |
|
The issue still persists in version 2.3.0. |
|
You agree this is a scipy issue, not a nibabel issue? |
|
I'm not entirely sure I'm afraid. There's a comment in the referenced scipy issue that speaks about the netcdf specification, and my knowledge of that is very very limited |
|
I just mean that nibabel's copy does not raise this error, and I don't think you're suggesting that we should make nibabel do that, so the discussion should all be over at scipy. Or, do you disagree? |
|
Yes, sure. If the test doesn't fail in your CI, can you please let us know what version of scipy/numpy it uses. nibabel 2.3.0 and scpiy 1.1.0/numpy 1.14.5 cause this error and these appear to be their latest versions. |
|
The tests don't fail because we ship out item version of this file.
…On Fri, 22 Jun 2018, 12:57 Ankur Sinha, ***@***.***> wrote:
Yes, sure. If the test doesn't fail in your CI, can you please let us know
what version of scipy/numpy it uses. nibabel 2.3.0 and scpiy 1.1.0/numpy
1.14.5 cause this error and these appear to be their latest versions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#579 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEIHDgpcXY2RuBR0O3fzq5e_rrBTLuFks5t_NvHgaJpZM4Qa115>
.
|
|
Sorry - bad typing on the phone. We vendor a copy of |
|
Ah, that makes sense. Should users also be suggested to use your copy then? I was simply using |
|
That's odd, that it fails with pip, because we do, by default, use our own copy, and that should not cause the error. I just tried tests locally, they do not fail. Can you give more details about your setup? |
|
That was me being dense. It does work in a fresh virtual env. I'll bisect scipy to see if I can narrow down when this regression was introduced. Thanks for the help. I reckon you can close the issue here if you want, or leave it open as a tracker while things get sorted out at the scipy end. |
|
I reached out to the netcdf folks for help here. Quoting the relevant bits from the most recent post: I expect the nibabel netcdf code handles this situation, while the scipy.io version does not. Looking at the code now. I'm not sure what the right way of handling this is: should one flag the file as corrupt and not parse it, or should one parse it and assume that the offending variable has a size of 0 (or something similar?). I'll also ask the netcdf folks for suggestions on the thread once I've looked at the code etc. |
|
I dropped a comment on the scipy ticket. I thought I'd drop one here too for others that may run into this later to see. The MNIC files do not seem to be standard netcdf (or they extend netcdf?), they cannot be read by the latest standard netcdf tools (as the netcdf folks found when they tried to investigate the issue). Nibable includes custom tweaks to read them properly (this PR I reckon), but currently scipy does not include these. I think it'll be OK to include them in scipy too if these tweaks do not affect how standard netcdf files are read in anyway. Thanks for the help. I expect we'll be able to tweak the Fedora package one way or another to get it to build properly depending on how the discussion on scipy goes. |
|
Hello from netcdf-land. This issue got my attention. The file The netcdf-C library was upgraded some time recently to perform more consistency checks on internal file pointers. This is probably the source of these recent nibabel test failures. I used netcdf low-level utilities to "repair" the internal pointer problem. I used a simple ncdump 4.4.1.1/ncgen sequence to preserve the original data contents. The repaired version is here. Note that only three bytes changed. The scalar integer variable Please replace the broken file version in your test suite with this repaired version, and see if this cleans up the test failures. I suspect the message I do not otherwise suspect anything wrong with nibabel test code or scipy. Software should not be extended or tweaked to attempt to read files like this with format violations. Such attempts may be dangerous and result in hidden data corruption. The other three MINC1 files within |
This happens when I use numpy-1.13.3 and scipy-1.0.0.
I know that nibabel bundles that part of scipy, but I need to unbundle all libraries when packaging in Fedora.
The text was updated successfully, but these errors were encountered: