Python runtime is not yet available on OS X #2312

Open
BanzaiMan opened this Issue May 14, 2014 · 50 comments

Comments

Projects
None yet
@BanzaiMan
Member

BanzaiMan commented May 14, 2014

No version of Python is available via virtualenv on OS X workers.

https://travis-ci.org/BanzaiMan/travis_production_test/builds/25122306

@BanzaiMan BanzaiMan added mac labels May 14, 2014

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan May 14, 2014

Member

The version available on Mavericks is 2.7.5.

Member

BanzaiMan commented May 14, 2014

The version available on Mavericks is 2.7.5.

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk May 14, 2014

Member

we should install virtualenv like we do with the linux workers, but talk to the great @dstufft about how we might install prebuilt python binaries on demand? :)

Member

joshk commented May 14, 2014

we should install virtualenv like we do with the linux workers, but talk to the great @dstufft about how we might install prebuilt python binaries on demand? :)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 14, 2014

I'm not aware of anything like that existing currently. I had some sketches for something like that but I don't have near the time to do that.

dstufft commented May 14, 2014

I'm not aware of anything like that existing currently. I had some sketches for something like that but I don't have near the time to do that.

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk May 14, 2014

Member

maybe we can talk more about your ideas, and what you think the time commitment would be to build it, and then we can see how we can make it happen?

Member

joshk commented May 14, 2014

maybe we can talk more about your ideas, and what you think the time commitment would be to build it, and then we can see how we can make it happen?

@BanzaiMan BanzaiMan added build and removed cookbooks labels May 14, 2014

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 14, 2014

I'm happy to discuss them, but it's unlikely there'll will be much that'll give me enough time to build it. I'm running on fumes as it is.

dstufft commented May 14, 2014

I'm happy to discuss them, but it's unlikely there'll will be much that'll give me enough time to build it. I'm running on fumes as it is.

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk May 14, 2014

Member

oh, no no no, i don't expect you to build it, but thinking maybe we can help fund it or something?

Member

joshk commented May 14, 2014

oh, no no no, i don't expect you to build it, but thinking maybe we can help fund it or something?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 14, 2014

Ah. Well I'm happy to advise or whatever :) It's not a very complicated thing I don't think. Fundamentally you just need a way to map an environment and maybe compile options to an identifier that you can use to download and unpack.

dstufft commented May 14, 2014

Ah. Well I'm happy to advise or whatever :) It's not a very complicated thing I don't think. Fundamentally you just need a way to map an environment and maybe compile options to an identifier that you can use to download and unpack.

@schlamar

This comment has been minimized.

Show comment
Hide comment
@schlamar

schlamar May 15, 2014

crypthography has a recipe to install Python on Travis OSX workers: https://github.com/pyca/cryptography/blob/master/.travis/install.sh#L29

crypthography has a recipe to install Python on Travis OSX workers: https://github.com/pyca/cryptography/blob/master/.travis/install.sh#L29

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 15, 2014

It's not actually a C application, it just pretends to be one because Python + OSX on travis-ci means builds error before they execute anything that a project has control over.

dstufft commented May 15, 2014

It's not actually a C application, it just pretends to be one because Python + OSX on travis-ci means builds error before they execute anything that a project has control over.

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan May 15, 2014

Member

Sure, but, as far as Travis is concerned, it is a C application. As a result, certain things do not happen for cryptography by default: setting $TRAVIS_PYTHON_VERSION, python key in .travis.yml (if it existed) is not used for constructing the build matrix, etc. If one chooses this route, one is responsible for setting up the sane environment for CI.

Member

BanzaiMan commented May 15, 2014

Sure, but, as far as Travis is concerned, it is a C application. As a result, certain things do not happen for cryptography by default: setting $TRAVIS_PYTHON_VERSION, python key in .travis.yml (if it existed) is not used for constructing the build matrix, etc. If one chooses this route, one is responsible for setting up the sane environment for CI.

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan May 15, 2014

Member

…which, I guess, is the part about "recipe" that @schlamar mentioned.

Member

BanzaiMan commented May 15, 2014

…which, I guess, is the part about "recipe" that @schlamar mentioned.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

The cryptography install script mentioned by @schlamar is using the pyenv (not to be confused with the pyvenv command that comes built-in with Python 3.3+) to build and install Python itself from source and manage shims to switch between Python versions. Using a custom built version of Python will prevent the users to pip install .whl packages with the macosx_10_6_intel.whl suffix.

Instead I would recommend using the official prebuilt .dmg archives from https://www.python.org/downloads/ that contain .mpkg installers. It is possible to script the install of such .dmg + .mpkg installers under OSX with a shell script such as the following:

http://blog.smalleycreative.com/administration/automating-osx-part-one/

Once official Python binaries are installed on OSX, it is possible to install pip with

curl -O https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py
sudo /Library/Frameworks/Python.framework/Versions/3.3/bin/python get-pip.py
sudo /Library/Frameworks/Python.framework/Versions/3.3/bin/pip install virtualenv
mkdir -p ~/virtualenv/
/Library/Frameworks/Python.framework/Versions/3.3/bin/virtualenv ~/virtualenv/python3.3
. ~/virtualenv/python3.3/bin/activate

Note: Python 3.4 (and later) comes with pip and pyvenv (a replacement for virtualenv) installed by default, so there is no need to use the get-pip.py bootstrap and the previous reduces to:

mkdir -p ~/virtualenv/
/Library/Frameworks/Python.framework/Versions/3.4/bin/pyvenv ~/virtualenv/python3.4
. ~/virtualenv/python3.4/bin/activate

I can help you working on such a travis setup for Python under OSX if you wish.

ogrisel commented May 16, 2014

The cryptography install script mentioned by @schlamar is using the pyenv (not to be confused with the pyvenv command that comes built-in with Python 3.3+) to build and install Python itself from source and manage shims to switch between Python versions. Using a custom built version of Python will prevent the users to pip install .whl packages with the macosx_10_6_intel.whl suffix.

Instead I would recommend using the official prebuilt .dmg archives from https://www.python.org/downloads/ that contain .mpkg installers. It is possible to script the install of such .dmg + .mpkg installers under OSX with a shell script such as the following:

http://blog.smalleycreative.com/administration/automating-osx-part-one/

Once official Python binaries are installed on OSX, it is possible to install pip with

curl -O https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py
sudo /Library/Frameworks/Python.framework/Versions/3.3/bin/python get-pip.py
sudo /Library/Frameworks/Python.framework/Versions/3.3/bin/pip install virtualenv
mkdir -p ~/virtualenv/
/Library/Frameworks/Python.framework/Versions/3.3/bin/virtualenv ~/virtualenv/python3.3
. ~/virtualenv/python3.3/bin/activate

Note: Python 3.4 (and later) comes with pip and pyvenv (a replacement for virtualenv) installed by default, so there is no need to use the get-pip.py bootstrap and the previous reduces to:

mkdir -p ~/virtualenv/
/Library/Frameworks/Python.framework/Versions/3.4/bin/pyvenv ~/virtualenv/python3.4
. ~/virtualenv/python3.4/bin/activate

I can help you working on such a travis setup for Python under OSX if you wish.

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan May 16, 2014

Member

If possible, we should stick with a uniform way of managing multiple Python (or any other language) runtimes on all platforms. Supporting different ways sounds appealing, but, in my opinion, support overhead will outweigh the "native" feel benefit in the long run.

Member

BanzaiMan commented May 16, 2014

If possible, we should stick with a uniform way of managing multiple Python (or any other language) runtimes on all platforms. Supporting different ways sounds appealing, but, in my opinion, support overhead will outweigh the "native" feel benefit in the long run.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

@BanzaiMan as far as I know, under Linux, python is installed with a ppa ( https://launchpad.net/~fkrull/+archive/deadsnakes ), hence this method is not possible under OSX. Using the official python.org binary packages for OSX sounds like the most standard way to install Python under OSX and is pretty uniform for all Python versions.

ogrisel commented May 16, 2014

@BanzaiMan as far as I know, under Linux, python is installed with a ppa ( https://launchpad.net/~fkrull/+archive/deadsnakes ), hence this method is not possible under OSX. Using the official python.org binary packages for OSX sounds like the most standard way to install Python under OSX and is pretty uniform for all Python versions.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 16, 2014

Linux uses pyenv other than for the system site packages variety.

dstufft commented May 16, 2014

Linux uses pyenv other than for the system site packages variety.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

Indeed I just saw the history of the CI env for linux: https://github.com/travis-ci/travis-cookbooks/commits/master/ci_environment/python . The deadsnakes PPA is no longer used since 2 months.

Where is the configuration of the OSX worker image? Is it also managed via the same chef recipes, or is there a specific infrastructure for OSX workers?

@dstufft what is your opinion on using python.org vs pyenv built python under OSX and the impact it has on distutils.util.get_platform?

On Python 3.4 installed from the dmg of python.org I get:

Python 3.4.0 (v3.4.0:04f714765c13, Mar 15 2014, 23:02:41)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.util import get_platform
>>> get_platform()
'macosx-10.6-intel'

On Python 3.4 installed using pyenv, I get:

Python 3.4.0 (default, May 16 2014, 14:36:29)
[GCC 4.7.3] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.util import get_platform
>>> get_platform()
'macosx-10.9-x86_64'

apparently this has an impact on the platform tags when I run the command pip wheel to build a binary distribution and the whl packages accepted when running pip install.

For instance if I run pip install numpy on the latter Python 3.4 (the one from pyenv), it will try to build numpy from the source tarball instead of reusing the official whl package provided by the numpy developers at: https://pypi.python.org/pypi/numpy

If I use the Python 3.4 of python.org, then pip install numpy directly installs the official whl package from pypi without trying to compile numpy from source (which is much faster and user friendlier).

I am not sure whether this is a problem in pip, wheel or a fundamental incompatibility in the 2 builds of the Python interpreter.

I think the default travis-ci should make it easy for python project maintainers to build wheels that they can upload to pypi for their OSX users.

I think that naive Python OSX users are quite likely to use the python installer from Python.org, hence my concerns.

ogrisel commented May 16, 2014

Indeed I just saw the history of the CI env for linux: https://github.com/travis-ci/travis-cookbooks/commits/master/ci_environment/python . The deadsnakes PPA is no longer used since 2 months.

Where is the configuration of the OSX worker image? Is it also managed via the same chef recipes, or is there a specific infrastructure for OSX workers?

@dstufft what is your opinion on using python.org vs pyenv built python under OSX and the impact it has on distutils.util.get_platform?

On Python 3.4 installed from the dmg of python.org I get:

Python 3.4.0 (v3.4.0:04f714765c13, Mar 15 2014, 23:02:41)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.util import get_platform
>>> get_platform()
'macosx-10.6-intel'

On Python 3.4 installed using pyenv, I get:

Python 3.4.0 (default, May 16 2014, 14:36:29)
[GCC 4.7.3] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.util import get_platform
>>> get_platform()
'macosx-10.9-x86_64'

apparently this has an impact on the platform tags when I run the command pip wheel to build a binary distribution and the whl packages accepted when running pip install.

For instance if I run pip install numpy on the latter Python 3.4 (the one from pyenv), it will try to build numpy from the source tarball instead of reusing the official whl package provided by the numpy developers at: https://pypi.python.org/pypi/numpy

If I use the Python 3.4 of python.org, then pip install numpy directly installs the official whl package from pypi without trying to compile numpy from source (which is much faster and user friendlier).

I am not sure whether this is a problem in pip, wheel or a fundamental incompatibility in the 2 builds of the Python interpreter.

I think the default travis-ci should make it easy for python project maintainers to build wheels that they can upload to pypi for their OSX users.

I think that naive Python OSX users are quite likely to use the python installer from Python.org, hence my concerns.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

I just found this interesting reference on platform tags, pip wheels and OSX: https://github.com/MacPython/wiki/wiki/Spinning-wheels

ogrisel commented May 16, 2014

I just found this interesting reference on platform tags, pip wheels and OSX: https://github.com/MacPython/wiki/wiki/Spinning-wheels

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 16, 2014

My plan was always to have travis on OSX not use the Python.org installer.

We'll probably at some point fix pip so that a 10.9 OSX will load something with the 10.6 SDK as that thing you linked said.

dstufft commented May 16, 2014

My plan was always to have travis on OSX not use the Python.org installer.

We'll probably at some point fix pip so that a 10.9 OSX will load something with the 10.6 SDK as that thing you linked said.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

So having read the above reference and @minrk's PR at pypa/pip#1465, it seems that the only way to build fat multi-arch wheels (with the 10_6_intel suffix) is to use a Python built on OSX 10.6 such as those distributed by python.org.

ogrisel commented May 16, 2014

So having read the above reference and @minrk's PR at pypa/pip#1465, it seems that the only way to build fat multi-arch wheels (with the 10_6_intel suffix) is to use a Python built on OSX 10.6 such as those distributed by python.org.

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel May 16, 2014

We'll probably at some point fix pip so that a 10.9 OSX will load something with the 10.6 SDK as that thing you linked said.

Which version of the OSX SDK is deployed on travis? If this is 10.9 or later, it means that wheels built on travis with a python built with the same SDK will not safely run on prior versions of OSX, no?

ogrisel commented May 16, 2014

We'll probably at some point fix pip so that a 10.9 OSX will load something with the 10.6 SDK as that thing you linked said.

Which version of the OSX SDK is deployed on travis? If this is 10.9 or later, it means that wheels built on travis with a python built with the same SDK will not safely run on prior versions of OSX, no?

@michaeljoseph michaeljoseph referenced this issue in audreyr/cookiecutter Jul 17, 2014

Closed

Try out travis-ci multi-os for osx #212

@lekhajee lekhajee referenced this issue in rackerlabs/mimic Jan 5, 2015

Closed

Py2app 88 #103

@ogrisel

This comment has been minimized.

Show comment
Hide comment
@ogrisel

ogrisel Apr 11, 2015

In the mean time, you might be interested in @matthew-brett's https://github.com/MacPython/terryfy to have bash utilities to install either Python.org, homebrew or macports installation of Python in a travis-ci OSX environment.

ogrisel commented Apr 11, 2015

In the mean time, you might be interested in @matthew-brett's https://github.com/MacPython/terryfy to have bash utilities to install either Python.org, homebrew or macports installation of Python in a travis-ci OSX environment.

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed Apr 15, 2015

Any progress in this regard?

I am seeing the following errors for the (changed) Neovim builds:

With Python 2.7 (https://travis-ci.org/neovim/neovim/jobs/58621371):

$ source ~/virtualenv/python["2.7"]/bin/activate
/Users/travis/build.sh: line 41: /Users/travis/virtualenv/python[2.7]/bin/activate: No such file or directory

Here the ["2.7"] seems odd.

And for Python 3.4 (https://travis-ci.org/neovim/neovim/jobs/58621375):

No output has been received in the last ... minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

blueyed commented Apr 15, 2015

Any progress in this regard?

I am seeing the following errors for the (changed) Neovim builds:

With Python 2.7 (https://travis-ci.org/neovim/neovim/jobs/58621371):

$ source ~/virtualenv/python["2.7"]/bin/activate
/Users/travis/build.sh: line 41: /Users/travis/virtualenv/python[2.7]/bin/activate: No such file or directory

Here the ["2.7"] seems odd.

And for Python 3.4 (https://travis-ci.org/neovim/neovim/jobs/58621375):

No output has been received in the last ... minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

@Juanlu001 Juanlu001 referenced this issue in poliastro/poliastro May 9, 2015

Closed

Added OS X testing to Travis CI #44

hkpeprah added a commit to square/pylink that referenced this issue Oct 10, 2017

[Support] Python 3 (#3)
* Add support for Python 3

* Test pylink against Python 2.7, 3.6, and 3.7-dev on osx and linux

* Disable osx Travis build as not fully supported yet.

See issue #2312 on the Travis-CI issue tracker (travis-ci/travis-ci#2312).

Anthchirp added a commit to Anthchirp/python-coldwallet that referenced this issue Oct 11, 2017

@scivision

This comment has been minimized.

Show comment
Hide comment
@scivision

scivision Oct 16, 2017

At least, if @travis-ci could have the error message state "Python is not supported on OSX images directly" that would save probably thousands of cumulative hours of user frustration before they realize they have to do a workaround

At least, if @travis-ci could have the error message state "Python is not supported on OSX images directly" that would save probably thousands of cumulative hours of user frustration before they realize they have to do a workaround

@BanzaiMan

This comment has been minimized.

Show comment
Hide comment
@BanzaiMan

BanzaiMan Oct 16, 2017

Member

Or, reject the build request altogether.

Member

BanzaiMan commented Oct 16, 2017

Or, reject the build request altogether.

@oyamad oyamad referenced this issue in QuantEcon/QuantEcon.py Oct 23, 2017

Open

Add OSX to travis.yml configuration #359

@moscoquera moscoquera referenced this issue in JMSwag/PyUpdater Oct 26, 2017

Closed

Travis: Run tests on macOS #92

@MichaelAquilina MichaelAquilina referenced this issue in MichaelAquilina/S4 Nov 1, 2017

Closed

Test OSX on Travis #91

@traumschule traumschule referenced this issue in traumschule/tomboy Nov 23, 2017

Closed

Test on osx #1

@nerdvegas nerdvegas referenced this issue in nerdvegas/rez Nov 28, 2017

Merged

Added Travis CI config #338

CAD97 added a commit to CAD97/rustfmt that referenced this issue Dec 4, 2017

Fix macOS build
- Ensure python is available travis-ci/travis-ci#2312
- Set PATH in a os-agnostic manner huonw/travis-cargo#71

CAD97 added a commit to CAD97/rustfmt that referenced this issue Dec 4, 2017

Fix macOS build
- Ensure python is available travis-ci/travis-ci#2312
- Only pip install --user on linux, macOS is in virtualenv
- Set PATH in a os-agnostic manner huonw/travis-cargo#71

@CAD97 CAD97 referenced this issue in rust-lang-nursery/rustfmt Dec 4, 2017

Merged

[Travis] Fix python/travis-cargo on macOS #2229

gsnedders added a commit to gsnedders/travis-macos-safari-test that referenced this issue Dec 6, 2017

@casperdcl casperdcl referenced this issue in CCPPETMR/SIRF Dec 13, 2017

Merged

Unify #71

5 of 12 tasks complete

gf712 added a commit to gf712/PyML that referenced this issue Dec 14, 2017

Xuanwo added a commit to yunify/qsctl that referenced this issue Dec 15, 2017

Xuanwo added a commit to yunify/qsctl that referenced this issue Dec 15, 2017

shirosaki added a commit to plasticboy/vim-markdown that referenced this issue Dec 15, 2017

@zfrenchee zfrenchee referenced this issue in joerick/cibuildwheel Dec 30, 2017

Closed

pip: command not found #48

@bjfultn bjfultn referenced this issue in California-Planet-Search/radvel Feb 4, 2018

Closed

Run Travis tests in Mac OS environment #136

benmwebb added a commit to ihmwg/python-ihm that referenced this issue Mar 4, 2018

Disable Mac Travis builds again
Turns out Travis doesn't support Python on Mac yet,
as per travis-ci/travis-ci#2312.

mishaturnbull added a commit to mishaturnbull/PySpeedTest that referenced this issue Mar 8, 2018

romainrichard added a commit to hacksaurz/nim that referenced this issue Mar 11, 2018

mikakoi added a commit to Codaone/DEXBot that referenced this issue Mar 12, 2018

Remove osx building from .travis.yml
Travis currently doesn't support python on mac. See travis-ci/travis-ci#2312

niemasd added a commit to niemasd/FAVITES that referenced this issue Mar 14, 2018

Remove Mac OS X from Travis CI
Travis currently doesn't support Python on Mac OS X. See travis-ci/travis-ci#2312

romainrichard added a commit to hacksaurz/nim that referenced this issue Mar 15, 2018

NIM-25 Moar pythons (#14)
Run nim on Python 3.6, 3.7-dev and pypy3.

Unfortunately Python isn't yet supported on OSX yet on Travis, so we can only ask for Linux
See travis-ci/travis-ci#2312 for details.

lemoney pushed a commit to lemoney/grease that referenced this issue Mar 20, 2018

Removal of OSX due to travis support: travis-ci/travis-ci#2312
Signed-off-by: Jimmy <contact@travelerbell.net>

lemoney pushed a commit to lemoney/grease that referenced this issue Mar 20, 2018

Removal of OSX due to travis support: travis-ci/travis-ci#2312
Signed-off-by: Jimmy <contact@travelerbell.net>

diegozea added a commit to PhyloSofS-Team/PhyloSofS that referenced this issue Mar 29, 2018

benjspriggs added a commit to PDXCapstoneF/SPECtate that referenced this issue Apr 3, 2018

@travis-ci travis-ci locked and limited conversation to collaborators Apr 8, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.