Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Python 2.x and 3.x #18588

Merged
merged 2 commits into from

6 participants

@samueljohn

Introduction

To avoid having multiple formulae with endings -py2 and -py3, we will handle support for different pythons (2.x vs. 3.x) in the same formula. Further brewed vs. external python will be transparently supported.

New python Requirements:

depends_on :python => :recommended
depends_on :python3 => :optional  # so --with-python3 option will be generated
# or to declare a specific version:
depends_on PythonInstalled.new("2.6")  

Most formulae should just work with this little change.

The python object that will support formula dev

In the Formula class there will be a python object available (python2 and python3, too), that supports some convenience methods which do the complicated work of figuring out the information no matter which python version or brewed vs. python from OS X:

python.site_packages # the site-packages in the Cellar for the current formulae that will later be linked globally
python.global_site_packages
python.binary # the full path to the python binary. For brewed python: HOMEBREW_PREFIX/opt/python/bin/python2
python.prefix
python.version
python.version.major
python.version.minor
python.xy # => e.g. "python2.7"
python.incdir # includes of python
python.libdir # the python dylib library
python.pkg_config_path # used internally by brew
python.from_osx?
python.framework?
python.universal?
python.pypy?
python.standard_caveats # The usual text to set PYTHONPATH that is only shown for python.from_osx?
python.if3then3 # => "" for 2.x and to "3" for 3.x.

Instead of using python, there is also python2 and python3 to explicitly refer to 2.x and 3.x resp..

Executing code only for 2.x or 3.x

It is also possible to

if python2
  # do something specific for python 2.x
end

if python3
  # do something specific for python 3.x
end

unless python
  # do something if there is no python (or python support was disabled --without-python --without-python3)
end

The python do ... end block

And one of the most important features is the python-block which creates the python.site_packages dir and already sets the PYTHONPATH correctly. This block is executes once for each python requirement.

Formulae that use a setup.py to install python bindings, can do

depends_on :python
depends_on :python3

python do
  # sets up PYTHONPATH and site-packages.
  # The python object within this block points to the python 2.x in the first round
  # and to 3.x in the second round.
  system python, "setup.py", "install", "--prefix=#{prefix}"
end

Are now all formulae automatically available for python 3.x?

No.

Adding Python 3.x support for individual formulae and testing that will take longer and I won't do this right now for all of the formulae. However, I did this for some to test things out.

Tasks:

  • depends_on :python Requirement (python is a synonym for python2 here)
  • depends_on :python3 Requirement
  • Write the correct sitecustomize.py also if the requirement is used.
  • Improve sitecustomize.py so that multiple versions (and external) are better supported.
  • Write the distutils.cfg if the requirement is used.
  • Update Python formulae to reuse sitecustomize.py
  • Allow formulae to use setup.py --prefix=#{prefix} instead of the complicated stuff as is.
  • pip will install executable scripts to HOMEBREW_PREFIX (solves a few issues)
  • Provide python.standard_caveats
  • Add Python's pkg-config during build
  • Use a temporary build environment to not touch stuff outside of the python block
  • Provide python.global_site_packages
  • Provide python.binary to indirect the call to python
  • Replace which_python by python.xy
  • Provide python do block
  • Provide python2 do block which is only run if a python 2.x has been found and unless --without-python.
  • Provide python.incdir (sometimes needed for cmake based projects)
  • Provide python.libdir (sometimes needed for cmake based projects)
  • Provide python.version.major and python.version.minor
  • Provide python.if3then3 which evaluates to "" for 2.x and to "3" for 3.x.
  • Provide python.framework (is nil for non-framework builds)
  • Provide python.universal?
  • Provide python.pypy?
  • Provide python.brewed?
  • Provide python.from_osx?
  • test :hg download strategy
  • Write test_python.rb
  • Do we want brew create --python ?
  • Check that the user has not python -> python3 symlink, which is not recommended.
  • Update audit to tell people to use the blocks and requirements

Formulae in homebrew core to update accordingly (list created via grep for python):

  • bazaar
  • beecrypt
  • boost
  • boost149
  • bulk_extractor
  • bup
  • cantera
  • cmu-sphinxbase
  • collectd
  • cracklib
  • cvs2svn
  • denyhosts
  • diffuse
  • disco
  • discodex
  • distcc
  • duplicity
  • enchant
  • fail2ban
  • fontforge
  • fsh
  • gdal
  • git-cola
  • git
  • glade
  • gnome-doc-utils
  • google-app-engine
  • gpsd
  • grass
  • grc
  • gtk-doc
  • io
  • ldns
  • ledger
  • libcaca
  • libdnet
  • liblunar
  • libming
  • libplist
  • libpst
  • librets
  • libstfl
  • libxml2
  • lilypond
  • little-cms
  • llvm
  • lorem
  • macvim
  • mapnik
  • mapserver
  • mercurial
  • node
  • omni
  • open-babel
  • opencolorio
  • orfeo
  • oscats
  • owfs
  • pil
  • portmidi
  • postgres-xc
  • postgresql
  • py2cairo
  • pyexiv2
  • pygtk
  • pygtkglext
  • pygtksourceview
  • pyqt
  • pyqwt
  • pyside-tools
  • pyside
  • python
  • python3
  • qscintilla2
  • rabbitmq-c
  • rabbitmq
  • rdiff-backup
  • rubber
  • scons
  • shiboken
  • shocco
  • sickbeard
  • sip
  • subversion
  • swatchbooker
  • thrift
  • treeline
  • uwsgi
  • vim
  • vte
  • weechat
  • wireshark
  • wxmac
  • xapian
  • xchat
  • yasm
  • yle-dl
  • zbar
  • zookeeper
@samueljohn samueljohn was assigned
@samueljohn

In that other comment is a short demo: #16439 (comment) how the python block looks. I post this here because the diff in the PR is large.

@jacknagel
Collaborator

Regarding testing. It is fairly difficult to test things that are integrated with Formula due to the amount of things that are coupled to the filesystem. The best strategy is to push as much behavior as possible into external objects that can be tested in isolation. My suggestion would be to push the python code in formula.rb into a new class, and pass the formula object into it so you can still instance_eval stuff in the correct context, e.g.

def python(*args, &block)
  SomePythonThing.new(self, *args, &block)
end

From there it is much easier to write tests without dragging the formula machinery into it.

@samueljohn

I am learning from the best.

@jacknagel
Collaborator

I looked at this a bit more and I see that what I suggested isn't an exact solution, because python needs to respond differently based on block_given? and it keeps state. So it would have to be more along the lines of

def some_python_thing
  @some_python_thing ||= SomePythonThing.new
end

def python(*args, &block)
  some_python_thing.call(*args, &block)
end

so that the object is only constructed once. But the principle is still the same.

BTW, if we're ever online at the same time we should hop on IRC and chat about the requirements API, I have some ideas I want to run by you.

@qrider70

Should I still try out the steps you have outlined in #16439 or will it be better to wait until you have this ironed out? Just wanted to check before trying it out.

@samueljohn

@qrider70 you don't have to but I am thankful for feedback. I assume you know to to use git to pull and reset the original state of your homebrew - if not; don't pull this. What should work already is the (re-)installation of python, python3, sip and pyqt. I am working on the other formulae...

@MindTooth

You have depends_on :python, :python3, but python, python2 blocks. For me this seems strange. But might be that I don't get the logic ;) Should not python without a number always be the latest? Sorry to take up your time.

@samueljohn

@MindTooth thanks for asking.

I imagine the following:

  • depends_on :python and depends_on :python2 are synonyms. The reasoning is that the python and python2 executables are also synonyms.
  • python do is the generic Python block that will be run for each python defined by the depends_on.
  • python2 do is only run if a python 2.x is present.
  • python3 do is only run if a python 3.x is present.

It is disputable if depends_on should only allow "python2" and "python3".

This allows us to write

depends_on :python2
depends_on :python3

def install
  python do
    <stuff that should be done for 2.x and 3.x>
  end

  python2 do
    <2.x specific handling here>
  end

  python3 do
    <3.x specific handling here>
  end
end
@qrider70

Ok. I haven't used git to do that yet. I could learn. :) At this point, what you mention as working (python, python3, sip, and pyqt) is all that I need afaik. Is there a way to use what you have to get just that, or is it better that I just wait until you have this fully ready? I am flexible either way for now. Thanks for your work on this. :)

@qrider70

Any news?

@samueljohn

soonish :-)

@samueljohn

Some formulae provide python bindings but don't need further work during the make step. Therefore, I made the python do ... end block return false if no python binding was requested (either by default or by --without-python) and then its possible to write:

if python do 
  system 'cmake', *args # python environment set up
end
else
  system 'cmake', *args # without python
end

This is really the same as

if python { system 'cmake', *args }
else
  system 'cmake', *args
end

where the latter makes it a bit more clear that python takes a block.

@samueljohn

This takes me so long, because I have to rework all python-using formulae a little bit ...
Today, I made good progress by getting ten more formulae to work with this modification.

@freaky-m0

Is there any progress in PySide for Python3? I would love to see it! Good work so far! :-)

@samueljohn

@freaky-m0 my changes in the core (on which I am working since days) raise the need to adapt each formula that uses python. That takes a bit of time. Therefore, I concentrate to first enable python 2.x support (that is the status-quo, but with my new system). Then I will push the changes, and after that, I will care to see which packages work with python3 (sometimes I have to rename executable scripts etc.)

To answer your question, I have not yet added the --with-python3 option to pyside. But from the project website it should work. I don't expect big trouble there.

I got working alraedy python3 bindings for numpy, pillow (PIL), PyQt, PyQwt, sip. But not yet pyside.

@samueljohn

So with these changes, when you get the error
Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
it means probably that the formula needs a depends_on :python.

@samueljohn

rebased on current master. For the guys who want to try out.

@samueljohn

I am almost done with this.
Good news is that I got PySide (and Shiboken) working with 2.x and 3.x in parallel, because it was requested quite often.

I'll rebase and make the commits more beautiful and then push. Even if I succesfully built each and every of the above listed formulae on my Mac (successfully), I except some rough edges with other OS X versions and with system python in few cases. So I'll watch the tracker. Maintainers, please tag issues with "python". All: Please mention my name in Python related topics.

Then, I will update the Wiki and the formulae in homebrew/science (and homebrew/versions etc).
I guess posting to the mailing list would be good.

samueljohn added some commits
@samueljohn samueljohn Python 2.x and 3.x support
New `depends_on :python` Dependency.
New `depends_on :python3` Dependency.

To avoid having multiple formulae with endings -py2 and -py3,
we will handle support for different pythons (2.x vs. 3.x)
in the same formula.
Further brewed vs. external python will be transparently supported.

The formula also gets a new object `python`, which is false if
no Python is available or the user has disabled it. Otherwise
it is defined and provides several support methods:

python.site_packages # the site-packages in the formula's Cellar
python.global_site_packages
python.binary # the full path to the python binary
python.prefix
python.version
python.version.major
python.version.minor
python.xy # => e.g. "python2.7"
python.incdir # includes of python
python.libdir # the python dylib library
python.pkg_config_path # used internally by brew
python.from_osx?
python.framework?
python.universal?
python.pypy?
python.standard_caveats # Text to set PYTHONPATH for python.from_osx?
python.if3then3 # => "" for 2.x and to "3" for 3.x.

Further, to avoid code duplication, `python` takes an optional
block that is run twice if the formula defines depends_on
:python AND :python3.

python do
  system python, 'setup.py', "--prefix=#{prefix}"
end

Read more in the Homebrew wiki.
0b50110
@samueljohn samueljohn superenv: Work if build tool changes DEVELOPER_DIR
Some build systems still set the DEVELOPER_DIR to /Developer
and then nothing works any more (xcrun, xcodebuild etc.)
I am looking at you MacVim.
13321d5
@samueljohn samueljohn merged commit 13321d5 into Homebrew:master
@adamv

These methods need to go into a module.

Alright. Going to make a module around this.

@adamv

Requiring inside of methods is not considered good form (though I forget why).

Thanks for looking at this, Adam. I can move it to the top of the file.

@adamv let's perhaps ask @jacknagel about this? I saw that Formula#installed_version also has a require.

@samueljohn

The to_s is needed to enable system python, "setup.py" where python.to_s is called.

@samueljohn samueljohn referenced this pull request from a commit in zaheerm/homebrew
@zaheerm zaheerm Added pygobject3 for python 2. 75b3fda
@samueljohn samueljohn deleted the samueljohn:python_next branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 3, 2013
  1. @samueljohn

    Python 2.x and 3.x support

    samueljohn authored
    New `depends_on :python` Dependency.
    New `depends_on :python3` Dependency.
    
    To avoid having multiple formulae with endings -py2 and -py3,
    we will handle support for different pythons (2.x vs. 3.x)
    in the same formula.
    Further brewed vs. external python will be transparently supported.
    
    The formula also gets a new object `python`, which is false if
    no Python is available or the user has disabled it. Otherwise
    it is defined and provides several support methods:
    
    python.site_packages # the site-packages in the formula's Cellar
    python.global_site_packages
    python.binary # the full path to the python binary
    python.prefix
    python.version
    python.version.major
    python.version.minor
    python.xy # => e.g. "python2.7"
    python.incdir # includes of python
    python.libdir # the python dylib library
    python.pkg_config_path # used internally by brew
    python.from_osx?
    python.framework?
    python.universal?
    python.pypy?
    python.standard_caveats # Text to set PYTHONPATH for python.from_osx?
    python.if3then3 # => "" for 2.x and to "3" for 3.x.
    
    Further, to avoid code duplication, `python` takes an optional
    block that is run twice if the formula defines depends_on
    :python AND :python3.
    
    python do
      system python, 'setup.py', "--prefix=#{prefix}"
    end
    
    Read more in the Homebrew wiki.
  2. @samueljohn

    superenv: Work if build tool changes DEVELOPER_DIR

    samueljohn authored
    Some build systems still set the DEVELOPER_DIR to /Developer
    and then nothing works any more (xcrun, xcodebuild etc.)
    I am looking at you MacVim.
Something went wrong with that request. Please try again.