1.8.3 breaks numpy install on py3 (ValueError: underlying buffer has been detached) #359

Closed
y-p opened this Issue Nov 22, 2012 · 20 comments

Comments

Projects
None yet
6 participants

y-p commented Nov 22, 2012

The error is (from travis output)

2320
2321 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/ccompiler.py", line 458, in CCompiler_get_version
2322
2323 status, output = exec_command(version_cmd,use_tee=0)
2324
2325 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/exec_command.py", line 197, in exec_command
2326
2327 if _with_python and (0 or sys.__stdout__.fileno()==-1):
2328
2329ValueError: underlying buffer has been detached

I've bisected this to 5e09944,
That commit is broken for me (giving a similar error to #358) but with a small fix
from the following commit, I can consistently reproduce the problem.
The offending line in numpy has been around since 2005, nothing new.

98b8160 works for me (and includes the 3.3 fix, FYI, @michaelklishin (Travis team)).

It looks like a lot of users of travis are affected by this , since all builds depending on numpy
are failing with this error (that's just my guess though).

the pasted error is from 3.1 on travis, I get the same thing on my local box with 3.2

y-p commented Nov 23, 2012

reverting 5e09944 on top of git master also seems to clear the problem.

y-p commented Nov 23, 2012

it looks like the updated distribute_setup.py embedded in virtualenv.py is the culprit.

Contributor

jezdez commented Nov 23, 2012

It would of course be better to fix the distribute-setup.py.

Contributor

jezdez commented Nov 23, 2012

FWIW, I've reproduced the issue on Mac OS with 3.3.0. Not sure what's causing it though.

Contributor

pnasrat commented Nov 23, 2012

@jezdez Lets revert 5e09944 and do a point release. To not block the OSS communities using travis/numpy

Next time we should do better at an RC/call for testing.

y-p commented Nov 23, 2012

a quick point release would be great.

Contributor

pnasrat commented Nov 23, 2012

I can replicate by doing what pip is calling under the hood.

/Users/pnasrat/Development/virtualenv/ve/bin/python3.3 -c "
__file__ = '/Users/pnasrat/Development/virtualenv/ve/build/numpy/setup.py'
from setuptools.command import egg_info
import pkg_resources
import os
def replacement_run(self):
    self.mkpath(self.egg_info)
    installer = self.distribution.fetch_build_egg
    for ep in pkg_resources.iter_entry_points('egg_info.writers'):
        # require=False is the change we're making:
        writer = ep.load(require=False)
        if writer:
            writer(self, ep.name, os.path.join(self.egg_info,ep.name))
    self.find_sources()
egg_info.egg_info.run = replacement_run
exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))
" egg_info --egg-base pip-egg-info 
Contributor

pnasrat commented Nov 23, 2012

Testing with 0.6.28 distribute in virtualenv

./ve/bin/pip install --no-install numpy
cd ve/build/numpy

Use script above or https://gist.github.com/4136264 and added a pdb.set_trace() in numpy build

(Pdb)  sys.__stdout__.fileno()
*** ValueError: underlying buffer has been detached
(Pdb) sys.__stdout__
*** ValueError: underlying buffer has been detached
Contributor

pnasrat commented Nov 23, 2012

OK so looking at distribute/setuptools/dist.py I see

sys.stdout = io.TextIOWrapper(
                sys.stdout.detach(), encoding, errors, newline, line_buffering)

https://bitbucket.org/tarek/distribute/changeset/7c543c05cd042c48bfa3926baa4a1a5543398305

Contributor

pnasrat commented Nov 23, 2012

Renaming handle_display_options to xhandle_display_options to disable works around.

y-p commented Nov 23, 2012

strange. 0.6.28 was released in july, that commit is from october, how can you reproduce with .28?

Contributor

pnasrat commented Nov 23, 2012

Argh ignore the 0.6.28 comment - I couldn't reproduce with that, the later tests were with 0.6.30 and changing that. sorry for the confusing comments

Contributor

pnasrat commented Nov 23, 2012

@jezdez clearly distribute bug lets do a point release with head of develop (that has another fix in) with 5e09944 rolled back.

y-p commented Nov 24, 2012

For summary so far, see comment in numpy/numpy#2761

Contributor

g2p commented Nov 24, 2012

Good summary. I'll point out that the distribute io tweak is only on Python 3 so far (it was motivated by a recent PEP), and that using sys.__stdout__ without a fallback should be discouraged (the documentation mentions some problem cases).

I've sent a pull request for distribute to leave sys.stdout alone on utf-8 platforms, and one for numpy to leave sys.__stdout__ alone because it was accessed for a frivolous reason.
Our options are to revert virtualenv's distribute update (meh, older distribute has other wrinkles), patch the distribute tarball virtualenv bundles (meh, but okay for virtualenv users), or get team distribute to publish a patch release (my preferred option, fixes the problem for everyone not just virtualenv users).

y-p commented Nov 24, 2012

Travis team says it'll take release+2 days (max) for builds to start passing again. it's been two days
of 🔴 so far.

jezdez closed this in f9dafad Nov 25, 2012

certik commented Dec 4, 2012

It seems it's still not working on Travis, right? E.g. here from today:

https://travis-ci.org/numpy/numpy/builds/3497226

Please let me know asap , what is the resolution for Numpy to work with PyPi , i.e. which versions of PyPui and Numpy will not have this issue after installation.

@jperkin jperkin pushed a commit to joyent/pkgsrc that referenced this issue Dec 9, 2013

wiz Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <pypa/virtualenv#359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.
b663640

@jsonn jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Mar 12, 2014

wiz Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <pypa/virtualenv#359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.
dfbd35f

MechCoder referenced this issue in scikit-learn/scikit-learn Jul 29, 2014

Closed

ValueError: underlying buffer has been detached #3498

@jsonn jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Oct 11, 2014

wiz Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <pypa/virtualenv#359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.
47c5180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment