1.8.3 breaks numpy install on py3 (ValueError: underlying buffer has been detached)

[ghost]

The error is (from travis output)

2320
2321 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/ccompiler.py", line 458, in CCompiler_get_version
2322
2323 status, output = exec_command(version_cmd,use_tee=0)
2324
2325 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/exec_command.py", line 197, in exec_command
2326
2327 if _with_python and (0 or sys.__stdout__.fileno()==-1):
2328
2329ValueError: underlying buffer has been detached

I've bisected this to 5e09944,
That commit is broken for me (giving a similar error to #358) but with a small fix
from the following commit, I can consistently reproduce the problem.
The offending line in numpy has been around since 2005, nothing new.

98b8160 works for me (and includes the 3.3 fix, FYI, @michaelklishin (Travis team)).

It looks like a lot of users of travis are affected by this , since all builds depending on numpy
are failing with this error (that's just my guess though).

the pasted error is from 3.1 on travis, I get the same thing on my local box with 3.2

[ghost]

reverting 5e09944 on top of git master also seems to clear the problem.

[ghost]

it looks like the updated distribute_setup.py embedded in virtualenv.py is the culprit.

[jezdez]

It would of course be better to fix the distribute-setup.py.

[jezdez]

FWIW, I've reproduced the issue on Mac OS with 3.3.0. Not sure what's causing it though.

[pnasrat]

@jezdez Lets revert 5e09944 and do a point release. To not block the OSS communities using travis/numpy

Next time we should do better at an RC/call for testing.

[ghost]

a quick point release would be great.

[pnasrat]

I can replicate by doing what pip is calling under the hood.

/Users/pnasrat/Development/virtualenv/ve/bin/python3.3 -c "
__file__ = '/Users/pnasrat/Development/virtualenv/ve/build/numpy/setup.py'
from setuptools.command import egg_info
import pkg_resources
import os
def replacement_run(self):
    self.mkpath(self.egg_info)
    installer = self.distribution.fetch_build_egg
    for ep in pkg_resources.iter_entry_points('egg_info.writers'):
        # require=False is the change we're making:
        writer = ep.load(require=False)
        if writer:
            writer(self, ep.name, os.path.join(self.egg_info,ep.name))
    self.find_sources()
egg_info.egg_info.run = replacement_run
exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))
" egg_info --egg-base pip-egg-info 

[pnasrat]

Testing with 0.6.28 distribute in virtualenv

./ve/bin/pip install --no-install numpy
cd ve/build/numpy

Use script above or https://gist.github.com/4136264 and added a pdb.set_trace() in numpy build

(Pdb)  sys.__stdout__.fileno()
*** ValueError: underlying buffer has been detached
(Pdb) sys.__stdout__
*** ValueError: underlying buffer has been detached

[pnasrat]

OK so looking at distribute/setuptools/dist.py I see

sys.stdout = io.TextIOWrapper(
                sys.stdout.detach(), encoding, errors, newline, line_buffering)

https://bitbucket.org/tarek/distribute/changeset/7c543c05cd042c48bfa3926baa4a1a5543398305

[pnasrat]

Renaming handle_display_options to xhandle_display_options to disable works around.

[ghost]

strange. 0.6.28 was released in july, that commit is from october, how can you reproduce with .28?

[pnasrat]

@stefanholek fyi https://bitbucket.org/tarek/distribute/changeset/7c543c05cd042c48bfa3926baa4a1a5543398305 breaks virtualenv + pip install numpy

[pnasrat]

Argh ignore the 0.6.28 comment - I couldn't reproduce with that, the later tests were with 0.6.30 and changing that. sorry for the confusing comments

[pnasrat]

Filed distribute bug https://bitbucket.org/tarek/distribute/issue/334/fix-for-311-breaks-packages-that-use

[pnasrat]

@jezdez clearly distribute bug lets do a point release with head of develop (that has another fix in) with 5e09944 rolled back.

[ghost]

For summary so far, see comment in numpy/numpy#2761

[g2p]

Good summary. I'll point out that the distribute io tweak is only on Python 3 so far (it was motivated by a recent PEP), and that using sys.__stdout__ without a fallback should be discouraged (the documentation mentions some problem cases).

I've sent a pull request for distribute to leave sys.stdout alone on utf-8 platforms, and one for numpy to leave sys.__stdout__ alone because it was accessed for a frivolous reason.
Our options are to revert virtualenv's distribute update (meh, older distribute has other wrinkles), patch the distribute tarball virtualenv bundles (meh, but okay for virtualenv users), or get team distribute to publish a patch release (my preferred option, fixes the problem for everyone not just virtualenv users).

[ghost]

Travis team says it'll take release+2 days (max) for builds to start passing again. it's been two days
of 🔴 so far.

[certik]

It seems it's still not working on Travis, right? E.g. here from today:

https://travis-ci.org/numpy/numpy/builds/3497226

[ashwini111]

Please let me know asap , what is the resolution for Numpy to work with PyPi , i.e. which versions of PyPui and Numpy will not have this issue after installation.