New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: test_big_arrays (test_io.TestSavezLoad) on OS X + Python 3.3 #3858

Open
rgommers opened this Issue Oct 3, 2013 · 27 comments

Comments

Projects
None yet
@rgommers
Member

rgommers commented Oct 3, 2013

Reported by Piet van Oostrum on the mailing list against 1.8.0rc1 on OS X with Python 3.3:

======================================================================
ERROR: test_big_arrays (test_io.TestSavezLoad)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy/testing/decorators.py", line 146, in skipper_func
    return f(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy/lib/tests/test_io.py", line 149, in test_big_arrays
    np.savez(tmp, a=a)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy/lib/npyio.py", line 530, in savez
    _savez(file, args, kwds, False)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy/lib/npyio.py", line 589, in _savez
    format.write_array(fid, np.asanyarray(val))
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy/lib/format.py", line 417, in write_array
    fp.write(array.tostring('C'))
OSError: [Errno 22] Invalid argument
@rgommers

This comment has been minimized.

Member

rgommers commented Oct 3, 2013

I can reproduce this. Looks like a Python 3.x bug.

import os
import sys
import time


if sys.maxsize > 2**32:
    print('64-bit')
else:
    print('32-bit, exiting')
    sys.exit(0)

fname = 'write_large_bytestring.txt'
tmp = open(fname, 'wb')
try:
    L = (1 << 31) + 100000
    tmp.write(b'abc' * 2**32)
finally:
    tmp.close()
    os.remove(fname)
    print('Elapsed time: %s s' % time.clock())

The above works with Python 2.7 but not with 3.3:

$ python tmp3.py 
64-bit
Elapsed time: 7.896957 s

$ python3.3 tmp3.py 
64-bit
Elapsed time: 50.149956 s
Traceback (most recent call last):
  File "tmp3.py", line 16, in <module>
    tmp.write(b'abc' * 2**32)
OSError: [Errno 22] Invalid argument

$ ulimit
unlimited

Both Python are installed from the dmgs on python.org. I can't find an issue for this on bugs.python.org but IIRC the io module was completely rewritten.

@rgommers

This comment has been minimized.

Member

rgommers commented Oct 3, 2013

Test introduced in gh-2942.

@pv

This comment has been minimized.

Member

pv commented Oct 3, 2013

Or maybe this is another OSX I/O bug? Remember, OSX libc is buggy and has issues in fwrite/fread when dealing with data blocks close to 2**32, which we had to work around in tofile/fromfile...

@njsmith

This comment has been minimized.

Member

njsmith commented Oct 3, 2013

In any case, we obviously have to work around it by splitting up the write
into smaller chunks, right?

(Even if it's ultimately libc's fault, python should probably work around
it itself - possibly 2.7 had code to do this that got lost in the
transition.)
On 3 Oct 2013 10:03, "Pauli Virtanen" notifications@github.com wrote:

Or maybe this is another OSX I/O bug? Remember, OSX libc is buggy and has
issues in fwrite/fread when dealing with data blocks close to 2**32...


Reply to this email directly or view it on GitHubhttps://github.com//issues/3858#issuecomment-25607852
.

@pv

This comment has been minimized.

Member

pv commented Oct 3, 2013

Re: gh-574 and gh-2806 and gh-3473 The OSX version in question may be relevant, maybe it now fails rather than writing garbage like it did previously?

@pv

This comment has been minimized.

Member

pv commented Oct 3, 2013

Yes, we can work around it by chunking. Maybe this issue should also be forwarded to Python devs, so that they could also implement chunking themselves...

@rgommers

This comment has been minimized.

Member

rgommers commented Oct 3, 2013

Ah, forgot about those issues. Tried to test on my 10.6 machine, but there the same script just hangs. Could be due to the hardware though, it's an ancient machine.

@charris

This comment has been minimized.

Member

charris commented Oct 6, 2013

Is this a 1.8.0 blocker? I'm going to put it there just so it isn't forgotten, it can always be removed. Does anyone know if it works for Python 3.2?

@charris

This comment has been minimized.

Member

charris commented Oct 6, 2013

Also, IIRC, we've only chunked reads, it may be that a test that writes a large file is broken.

@rgommers

This comment has been minimized.

Member

rgommers commented Oct 6, 2013

I wouldn't hold up the release for this one, it's not a regression. Mark it knownfail though in the 1.8.x branch if it's not failed before the release.

@charris

This comment has been minimized.

Member

charris commented Oct 6, 2013

@rgommers Does if fail only with OSX and 3.3? You say that it works for python 2.7, is that correct?

@rgommers

This comment has been minimized.

Member

rgommers commented Oct 6, 2013

It doesn't fail for 2.7, but the test doesn't check that what's written to file is correct. It's likely not to be correct, see other issues that Pauli linked.

@charris

This comment has been minimized.

Member

charris commented Oct 6, 2013

@rgommers So OSX in general. I'll leave it open in 1.9-devel to motivate a fix and open an issue.

@charris

This comment has been minimized.

Member

charris commented Feb 24, 2014

This is reported fixed in OS X Mavericks. This is probably won't fix, as the proper fix is to upgrade the OS.

See also #2931.

@charris

This comment has been minimized.

Member

charris commented May 5, 2014

Closing, should be fixed by Mavericks. Please reopen if the problem persists.

@charris charris closed this May 5, 2014

@dougalsutherland

This comment has been minimized.

dougalsutherland commented Jul 8, 2014

This is still happening for me on Mavericks with numpy 1.8.1 and python 3.4 (also 3.3) from Anaconda; if I comment out the skipif decorator, https://github.com/numpy/numpy/blob/v1.8.1/numpy/lib/tests/test_io.py#L154 fails.

The test passes, and data appears to be loaded correctly, using python 2.7 from Anaconda.

This isn't necessarily a flaw with numpy, but others appear to be working around this or similar issues, e.g. torch/DEPRECEATED-torch7-distro@40e6593 (which is for reading rather than writing, but I'm also unable to read large files on python 3).

@AndreasMadsen

This comment has been minimized.

AndreasMadsen commented Jul 10, 2014

Running the testcase using Python 3.4 with numpy 1.8.1 on Mac OS X 10.9.4 (Mavericks) results in the known OSError: [Errno 22] Invalid argument error. @certik Please reopen issue!

@makmanalp

This comment has been minimized.

makmanalp commented Jul 20, 2015

Another data point - using OSX 10.9.5 (Mavericks) and I get the same issue. I just saw this bug in the python tracker: https://bugs.python.org/issue24658

@charris charris reopened this Jul 20, 2015

@charris

This comment has been minimized.

Member

charris commented Jul 20, 2015

Might as well reopen this. I don't know if it will be fixed when Python solves their part, but we will find out.

@adammenges

This comment has been minimized.

adammenges commented Apr 22, 2017

Happens here, latest macOS, python (3.6), and numpy.

@danlou

This comment has been minimized.

danlou commented Sep 26, 2017

FYI Still seeing this bug on latest macOS (10.12.6) and Python 3.5.2 (Anaconda 4.2.0).
Has anyone figured out an upper limit on chunk sizes?

@divyansha

This comment has been minimized.

divyansha commented Nov 28, 2017

Getting this error as well, macOS Sierra, python 3.6, numpy

@charris

This comment has been minimized.

Member

charris commented Nov 29, 2017

There looks to be some motion on the Python issue, but perhaps we should just go ahead and chunk the writes.

@charris charris modified the milestones: 1.9 blockers, 1.15.0 release Nov 29, 2017

@charris

This comment has been minimized.

Member

charris commented Jun 7, 2018

Looks like this isn't going to get fixed upstream anytime soon. Anyone know the latest status?

@charris charris removed this from the 1.15.0 release milestone Jun 7, 2018

@charris charris added this to the 1.16.0 release milestone Jun 7, 2018

@matrixise

This comment has been minimized.

matrixise commented Oct 19, 2018

Hi all,

Could you try this issue with the last version of Python 3.6, 3.7 and 3.8a because I think to have fixed the issue on OSX with this PR (python/cpython#1705).

I have an other PR for 2.7, but this one is not yet ready :/

Thank you for your feedback.

@charris

This comment has been minimized.

Member

charris commented Nov 17, 2018

@rgommers Any chance you can test this? Any other feedback on the current status of this would be welcome.

@rgommers

This comment has been minimized.

Member

rgommers commented Nov 19, 2018

The test_big_arrays test passes, but #3858 (comment) still fails for me with the latest Python 3.6 shipped by Anaconda. That probably doesn't have the CPython fix though. No time to build Python myself right now, sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment