New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.8.3 breaks numpy install on py3 (ValueError: underlying buffer has been detached) #359

Closed
ghost opened this Issue Nov 22, 2012 · 20 comments

Comments

Projects
None yet
5 participants
@ghost

ghost commented Nov 22, 2012

The error is (from travis output)

2320
2321 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/ccompiler.py", line 458, in CCompiler_get_version
2322
2323 status, output = exec_command(version_cmd,use_tee=0)
2324
2325 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/exec_command.py", line 197, in exec_command
2326
2327 if _with_python and (0 or sys.__stdout__.fileno()==-1):
2328
2329ValueError: underlying buffer has been detached

I've bisected this to 5e09944,
That commit is broken for me (giving a similar error to #358) but with a small fix
from the following commit, I can consistently reproduce the problem.
The offending line in numpy has been around since 2005, nothing new.

98b8160 works for me (and includes the 3.3 fix, FYI, @michaelklishin (Travis team)).

It looks like a lot of users of travis are affected by this , since all builds depending on numpy
are failing with this error (that's just my guess though).

the pasted error is from 3.1 on travis, I get the same thing on my local box with 3.2

@ghost

This comment has been minimized.

ghost commented Nov 23, 2012

reverting 5e09944 on top of git master also seems to clear the problem.

@ghost

This comment has been minimized.

ghost commented Nov 23, 2012

it looks like the updated distribute_setup.py embedded in virtualenv.py is the culprit.

@jezdez

This comment has been minimized.

Contributor

jezdez commented Nov 23, 2012

It would of course be better to fix the distribute-setup.py.

@jezdez

This comment has been minimized.

Contributor

jezdez commented Nov 23, 2012

FWIW, I've reproduced the issue on Mac OS with 3.3.0. Not sure what's causing it though.

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

@jezdez Lets revert 5e09944 and do a point release. To not block the OSS communities using travis/numpy

Next time we should do better at an RC/call for testing.

@ghost

This comment has been minimized.

ghost commented Nov 23, 2012

a quick point release would be great.

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

I can replicate by doing what pip is calling under the hood.

/Users/pnasrat/Development/virtualenv/ve/bin/python3.3 -c "
__file__ = '/Users/pnasrat/Development/virtualenv/ve/build/numpy/setup.py'
from setuptools.command import egg_info
import pkg_resources
import os
def replacement_run(self):
    self.mkpath(self.egg_info)
    installer = self.distribution.fetch_build_egg
    for ep in pkg_resources.iter_entry_points('egg_info.writers'):
        # require=False is the change we're making:
        writer = ep.load(require=False)
        if writer:
            writer(self, ep.name, os.path.join(self.egg_info,ep.name))
    self.find_sources()
egg_info.egg_info.run = replacement_run
exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))
" egg_info --egg-base pip-egg-info 
@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

Testing with 0.6.28 distribute in virtualenv

./ve/bin/pip install --no-install numpy
cd ve/build/numpy

Use script above or https://gist.github.com/4136264 and added a pdb.set_trace() in numpy build

(Pdb)  sys.__stdout__.fileno()
*** ValueError: underlying buffer has been detached
(Pdb) sys.__stdout__
*** ValueError: underlying buffer has been detached
@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

OK so looking at distribute/setuptools/dist.py I see

sys.stdout = io.TextIOWrapper(
                sys.stdout.detach(), encoding, errors, newline, line_buffering)

https://bitbucket.org/tarek/distribute/changeset/7c543c05cd042c48bfa3926baa4a1a5543398305

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

Renaming handle_display_options to xhandle_display_options to disable works around.

@ghost

This comment has been minimized.

ghost commented Nov 23, 2012

strange. 0.6.28 was released in july, that commit is from october, how can you reproduce with .28?

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

Argh ignore the 0.6.28 comment - I couldn't reproduce with that, the later tests were with 0.6.30 and changing that. sorry for the confusing comments

@pnasrat

This comment has been minimized.

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Nov 23, 2012

@jezdez clearly distribute bug lets do a point release with head of develop (that has another fix in) with 5e09944 rolled back.

@ghost

This comment has been minimized.

ghost commented Nov 24, 2012

For summary so far, see comment in numpy/numpy#2761

@g2p

This comment has been minimized.

Contributor

g2p commented Nov 24, 2012

Good summary. I'll point out that the distribute io tweak is only on Python 3 so far (it was motivated by a recent PEP), and that using sys.__stdout__ without a fallback should be discouraged (the documentation mentions some problem cases).

I've sent a pull request for distribute to leave sys.stdout alone on utf-8 platforms, and one for numpy to leave sys.__stdout__ alone because it was accessed for a frivolous reason.
Our options are to revert virtualenv's distribute update (meh, older distribute has other wrinkles), patch the distribute tarball virtualenv bundles (meh, but okay for virtualenv users), or get team distribute to publish a patch release (my preferred option, fixes the problem for everyone not just virtualenv users).

@ghost

This comment has been minimized.

ghost commented Nov 24, 2012

Travis team says it'll take release+2 days (max) for builds to start passing again. it's been two days
of 🔴 so far.

@jezdez jezdez closed this in f9dafad Nov 25, 2012

jezdez added a commit that referenced this issue Nov 25, 2012

Merge pull request #363 from g2p/develop
Update distribute to 0.6.31, includes a workaround for #359. Update changelog.
@certik

This comment has been minimized.

certik commented Dec 4, 2012

It seems it's still not working on Travis, right? E.g. here from today:

https://travis-ci.org/numpy/numpy/builds/3497226

@ashwini111

This comment has been minimized.

ashwini111 commented Apr 23, 2013

Please let me know asap , what is the resolution for Numpy to work with PyPi , i.e. which versions of PyPui and Numpy will not have this issue after installation.

jperkin pushed a commit to joyent/pkgsrc-legacy that referenced this issue Dec 9, 2013

wiz
Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <https://github.com/pypa/virtualenv/issues/359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Mar 12, 2014

wiz
Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <https://github.com/pypa/virtualenv/issues/359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Oct 11, 2014

wiz
Update to 0.6.32:
------
0.6.32
------

* Fix test suite with Python 2.6.
* Fix some DeprecationWarnings and ResourceWarnings.
* Issue #335: Backed out `setup_requires` superceding installed requirements
  until regression can be addressed.

------
0.6.31
------

* Issue #303: Make sure the manifest only ever contains UTF-8 in Python 3.
* Issue #329: Properly close files created by tests for compatibility with
  Jython.
* Work around Jython bugs `#1980 <http://bugs.jython.org/issue1980>`_ and
  `#1981 <http://bugs.jython.org/issue1981>`_.
* Issue #334: Provide workaround for packages that reference `sys.__stdout__`
  such as numpy does. This change should address
  `virtualenv #359 <https://github.com/pypa/virtualenv/issues/359>`_ as long
  as the system encoding is UTF-8 or the IO encoding is specified in the
  environment, i.e.::

     PYTHONIOENCODING=utf8 pip install numpy

* Fix for encoding issue when installing from Windows executable on Python 3.
* Issue #323: Allow `setup_requires` requirements to supercede installed
  requirements. Added some new keyword arguments to existing pkg_resources
  methods. Also had to updated how __path__ is handled for namespace packages
  to ensure that when a new egg distribution containing a namespace package is
  placed on sys.path, the entries in __path__ are found in the same order they
  would have been in had that egg been on the path when pkg_resources was
  first imported.

------
0.6.30
------

* Issue #328: Clean up temporary directories in distribute_setup.py.
* Fix fatal bug in distribute_setup.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment