Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

request: virtualenv on OSX unrequire Xcode #168

Closed
gregglind opened this Issue · 21 comments

9 participants

@gregglind

1) Xcode now costs money. (and a login at the app store, etc.)
2) venv is best practices
3) I want to teach people best practices right off the bat.

Could there be a solution that installs virtualenv, but doesn't complain about the xcode necessary bits until they are actually needed (i.e., until a c-extension needs to be compiled)? As an alternative, would it make sense to explore a 'minimum xcode install' alternative?

(This has actively bitten me in trying to write/run classes for PyStar.org)

@carljm
Owner

I don't use OSX myself, but my understanding is that Xcode actually is needed for virtualenv to work at all on OSX; it isn't about installing C extensions. It has to do with using the Apple framework install of Python, and needing to use the "install_name_tool" command (which, IIUC, comes only with Xcode) to tell the virtualenv python executable about its new location.

I guess if you compile your own Python and use that with virtualenv instead of the framework install, you might not need install_name_tool -- but if you've compiled your own Python you've clearly got Xcode anyway!

If you have more information here, or want to assert that install_name_tool is not in fact required, or is available without Xcode, please start a thread about it on the mailing list (http://groups.google.com/group/python-virtualenv) -- I'd like to get confirmation from some other OSX virtualenv users before making any changes here.

@carljm carljm closed this
@gregglind

Thanks for the fast response and the information! I couldn't quite remember the mechanism on this one, and my search skills were failing me!

@jezdez
Owner

For the record, Xcode for Lion is free (http://itunes.apple.com/us/app/xcode/id448457090?mt=12).

@carljm
Owner

So, talked to Glyph about this at DjangoCon, and he thinks it would be possible to do the same thing that we currently do with install_name_tool using instead the "macholib" library that is available on OSX Pythons. If we could make this change, it would mean Xcode would no longer be required to use virtualenv on OSX, which would be awesome. However, he didn't say it would be easy, so it will probably require an OSX user to devote some significant time to it. Reopening this issue to track this possibility.

@carljm carljm reopened this
@glyph

Just to clarify; I'm not completely sure about this :). Take a look at what py2app does when it's manipulating its stub binary to stick into an app bundle though, and you might be able to figure something out from there.

@mitsuhiko

Yes, macholib is a possibility but I think the easier and better solution would be to provide a compiled version of install_name_tool. It's only there for OS X anyways and it's a single executable not linked against anything but the system lib. It's open source under MIT or something as well.

Source: http://www.opensource.apple.com/source/cctools/cctools-750/misc/install_name_tool.c

@mitsuhiko

Jup, but that does not mean we have the right to redistribute these binaries I think. (IANAL)

@gregglind

Carljim, perhaps this will help?

from: http://jessenoller.com/2011/07/30/quick-pythondeveloper-tips-for-osx-lion/

https://github.com/kennethreitz/osx-gcc-installer

Side note: @kennethreitz, gen­tle­man scholar, has actu­ally done a cus­tom osx-gcc-installer — this con­tains a sys­tem install of GCC and all the good­ness you need (such as install_name_tool) for Python hack­ing. So you might be able to skip the mas­sive XCode install.

@danchr

@mitsuhiko: Nor am I, but I'd consider it likely that you do, given that install_name_tool is APSL. Besides, it's not that hard to build it from source…

@mitsuhiko

And the S stands for source. Unless I am terribly blind it does not mention binaries at all. Yes, you can build it yourself, but it would be a lot easier for virtualenv if we could just pull in the binaries from macosforge and ship them.

@danchr

The APSL was approved by both the Open Source Initiative and the Free Software Foundation, so it should give you ample rights to distribute binaries. The links I provided earlier were for Apple's DarwinSource project, which exists in order to allow other people build the base system. I suppose you could ask on darwinbuild-dev if you feel uncertain whether the roots are redistributable, or if it's okay to link to them…

@mitsuhiko

Assuming this is legally possible we could ship the binaries for install_name_tool (last three versions of OS X). A stupid simple compression would be concatenating all three files, remembering the offsets, zlib encoding them and base64ing them. This would come to around 300KB extra size for virtualenv. I think we need all since they support different architectures but I have not verified this.

Alternatively (and what I would think makes more sense) would be to provide a script that download install_name_tool and drops it into /usr/local/bin if it can't find install_name_tool.

Regarding licensing I think you can't be too cautious when it comes to apple.

@gregglind

Any additional traction on this? We are gearing up for the next Minneapolis PyStar workshop in 3 weeks, and it would be great to see something happen on this. The Xcode dependency is a major blocker for 'doing things right'.

GL

@gregglind gregglind referenced this issue from a commit in gregglind/virtualenv
@gregglind gregglind install_name_tool in OSX Framework builds #168
shouldfix:  creating a virtualenv from an OSX
Framework build should no longer require XCode to
run successfully.

implementaion:  if ``install_name_tool`` missing,
download binaries from macosxforge, and install
them to a temp directory

rejected:  download and compile .c source.

TODOS:

*   review for python2/3 issues
*   cleanup of tempfiles?  Assume the system will do it?
*   test for robustness
*   review implications of not having XCode for packages, if any
*   integrate with #54 pull or its descendents.  (trivial)

Possible Test Code:

    #!/bin/sh

    deactivate # virtualenv on system (Framework) python

    # assuming I have XCode or ``install_name_tool`` in path.
    python virtualenv.py t_install_name_tool  # creates with no issue
    t_install_name_tool/bin/python -c ""

    # remove it from path
    sudo mv /usr/bin/install_name_tool{,2}

    # should create with no issue, using tmpfile
    python virtualenv.py t_tmp_install_name_tool
    t_tmp_install_name_tool/bin/python -c ""

    # clean up
    sudo mv /usr/bin/install_name_tool{2,}
    command rm -rf t_{,tmp_}install_name_tool
a614215
@gorakhargosh gorakhargosh referenced this issue from a commit in gorakhargosh/virtualenv
@gregglind gregglind install_name_tool in OSX Framework builds #168
shouldfix:  creating a virtualenv from an OSX
Framework build should no longer require XCode to
run successfully.

implementaion:  if ``install_name_tool`` missing,
download binaries from macosxforge, and install
them to a temp directory

rejected:  download and compile .c source.

TODOS:

*   review for python2/3 issues
*   cleanup of tempfiles?  Assume the system will do it?
*   test for robustness
*   review implications of not having XCode for packages, if any
*   integrate with #54 pull or its descendents.  (trivial)

Possible Test Code:

    #!/bin/sh

    deactivate # virtualenv on system (Framework) python

    # assuming I have XCode or ``install_name_tool`` in path.
    python virtualenv.py t_install_name_tool  # creates with no issue
    t_install_name_tool/bin/python -c ""

    # remove it from path
    sudo mv /usr/bin/install_name_tool{,2}

    # should create with no issue, using tmpfile
    python virtualenv.py t_tmp_install_name_tool
    t_tmp_install_name_tool/bin/python -c ""

    # clean up
    sudo mv /usr/bin/install_name_tool{2,}
    command rm -rf t_{,tmp_}install_name_tool

The original patch was cleaned up, tested against
at Python 2.5, 2.6, 2.7, 3.2, and verified to work.

Signed-off-by: Yesudeep Mangalapilly <yesudeep@google.com>
1306777
@revolunet

+1 for this PR; we dont need that FAT xcode here

@glandium

We're having the issue at Mozilla, and we don't want to force everyone to have to install Xcode, especially when they don't need to deal with compiling code (localizers are such people).
The only reason I can see install_name_tool is necessary is because the python library is copied from the python framework into the virtualenv directory. What is that needed for? Why not use the python library from the python framework, and lift the need for install_name_tool altogether?
Alternatively, without using install_name_tool, it is possible to force python into using the python library copy by setting the DYLD_INSERT_LIBRARIES variable to point to the python library file. (e.g. DYLD_INSERT_LIBRARIES=$virtualenvdir/.Python)

@glandium

Answering myself: python derives sys.prefix from the library location, which explains why the library is copied.

@carljm
Owner

Pull request #212 / #241 is a really ugly approach that frankly I'd rather avoid.

IMO the most promising option discussed here is a pure-Python replacement for install_name_tool using macholib. It looks like @pnasrat already put some work into this (https://gist.github.com/935329) but I have at least one report (from @luser) that the macholib APIs it uses don't seem to match up with current versions of macholib. If someone wants to pick this up and verify that it works, and turn it into a pull request against virtualenv (with macholib, or even better the minimal necessary portions of it, bundled), that's an approach that I think would make everyone happy.

@gregglind

http://kennethreitz.com/xcode-gcc-and-homebrew.html might be the better solution, and the one I would probably recommend, even over my own patches. Lots of other useful libraries (Numpy, Fabric/Paramiko) need compilation, and explaining to users about 'what went wrong' is painful.

10.7 Xcode doesn't even come with GCC and environment (4.3!).

As a possible derail, does Rails / Ruby have this issue?

(note: most of us on this are Moz people right now, if we want to chat about it in-house :)

@jrand

I tried to make a macholib replacement for install_name_tool and the resulting binaries only worked on certain OS releases. I can't find my notes at the moment, but I'd guess it worked on 10.6, but failed on 10.7.

@carljm
Owner

Anyone interested in seeing this issue fixed - please test out pull request #289 and post your results there. The diff looks good to me and I'll likely merge it soon, pending any issues discovered in further testing.

@jezdez jezdez closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.