build breaks with new virtualenv 1.8.3 on py3 #2761

Closed
y-p opened this Issue Nov 22, 2012 · 16 comments

Comments

Projects
None yet
4 participants

y-p commented Nov 22, 2012

2320
2321 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/ccompiler.py", line 458, in CCompiler_get_version
2322
2323 status, output = exec_command(version_cmd,use_tee=0)
2324
2325 File "/home/travis/virtualenv/python3.1/build/numpy/build/py3k/numpy/distutils/exec_command.py", line 197, in exec_command
2326
2327 if _with_python and (0 or sys.__stdout__.fileno()==-1):
2328
2329ValueError: underlying buffer has been detached

I've opened a coressponding issue in venv: pypa/virtualenv#359.
If the numpy team could lend a hand in debugging this, that'd be great.

Contributor

g2p commented Nov 23, 2012

Here's the distribute followup: https://bitbucket.org/tarek/distribute/issue/334/

Paging @teoliphant and @pearu who may explain that part of the code.

Owner

rgommers commented Nov 23, 2012

This is not a numpy issue but a Python bug. Caused by:

$ python3.2
Python 3.2.3 (default, Oct 19 2012, 20:13:42) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.detach()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: underlying buffer has been detached

Same issue at statsmodels/statsmodels#204 and luispedro/mahotas#25.

Comment from statsmodels issue:
This was reported before, but apparently core devs don't approve of using detach: https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/QHyrZhN79D0
It's recommended in the Python docs though: http://docs.python.org/3.2/library/sys.html

The use of sys.stdout.detach in distribute should be undone of course, but the bug is a Python one.

@rgommers rgommers closed this Nov 23, 2012

Contributor

g2p commented Nov 23, 2012

detach seems like the right thing to do for distribute's requirement of having stdout in the right encoding. I was wondering if numpy had a particular need to communicate. Why does numpy code use sys.__stdout__ rather than sys.stdout?

Contributor

g2p commented Nov 23, 2012

Oh, to clarify: there's no bug in the way detach is used by distribute. sys.stdout is reassigned atomically. Quick demo:

python3 -c 'import sys; sys.stdout = sys.stdout.detach()'

@g2p g2p referenced this issue in statsmodels/statsmodels Nov 23, 2012

Closed

Cython extensions built twice? #204

y-p commented Nov 23, 2012

hmm, I don't know about that.

I believe the offending line in distribute's dist.py is:

sys.stdout = io.TextIOWrapper(sys.stdout.detach(), 'utf-8', 
                                                    errors, newline, line_buffering)

python is strict and the constructor is called prior to the assignment.

Contributor

g2p commented Nov 23, 2012

Sure, but the "detach() is broken" line of thought is still a red herring in this discussion. It's not what's happening here.

y-p commented Nov 23, 2012

comment removed, my bad

Owner

rgommers commented Nov 23, 2012

@g2p you're right that like that the detach call does not raise an error.

This code in numpy is almost 10 years old, written to support Python 2.1. Trying to find out what problem was being worked around at that time will be a painful and not very useful way to spend ones time.

Anyway this code is present in many released numpy versions; many other packages depend on numpy.distutils. The only reasonable solution I can see is to undo the distribute commit.

Owner

rgommers commented Nov 23, 2012

Or fix it by also replacing __stdout__ so it's still the same object as stdout.

y-p commented Nov 24, 2012

Summary so far (as I understand it):

  1. sys.__stdout__ is documented (2/3) in python as holding the value of stdout at the start of the program.
  2. Distribute merged a commit during the 6.30 cycle which invokes sys.stdout.detach() to
    handle some utf-8 issue on py3.
  3. Said commit's use of detach() invalidates sys.__stdout__ but not std.stdout.
  4. numpy uses sys.__stdout__ rather then sys.stdout in some legacy code from 2005,
    for a reason now probably lost in time. When that code references sys.__stdout__ the exception
    ValueError: underlying buffer has been detached is raised.
  5. 1.8.3 is the first point release of virtualenv to have working support for python 3.3.
  6. during the 1.8.3 cycle, virtualenv upgraded to distribute 6.30 as their embedded dependency (commit).
  7. Travis-ci seems to have been upgraded to 1.8.3 about two days ago in order to enable support for py3.3.
  8. Projects which support py3,x and depend on numpy.distutils (certainly all that depend on numpy) are currently experiencing failed runs on travis , Users that have upgraded to distribute 6.30 will experience install failures under py3.
Owner

rgommers commented Nov 24, 2012

Good summary. About 8), it's not only related to Travis, but anyone who installs distribute 0.6.30 won't be able to install things that rely on numpy.distutils I think.

y-p commented Nov 24, 2012

Updated, thank you.

Contributor

astrofrog commented Nov 29, 2012

Does anyone have a good workaround for Travis?

y-p commented Dec 4, 2012

For future reference: fixed in master branch by 649c908

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment