python goes superenv

- Install a sitecustomize.py that is only executed for brewed
  python to
  -  Fix the prefix, python thinks it is installed to.
     (Remember, Python thinks it lives in the Cellar)
  -  Remove "/System/..." stuff from sys.path which caused
     a lot of install trouble because setuptools has the
     habbit to inject itself upfront, overwriting our distribute.
- Allow --with-poll and don't say, we didn't warn you.
- Don't need depends_on :x11 any longer. Yeah, no XQuartz!
- Add --with-brewed-openssl
- pip 1.2.1
- pip, pip-2.7, easy_install and easy_install-2.7 are installed
  to prefix, such that they are directly available, even if
  people have not set their PATH to include
  $(brew --prefix)/share/python
- Caveats shorter and clear.
- For Xcode-only:
  - Patch the distutils buildsystem to use "xcrun cc" etc.
  - Teach distutils the MacOS.sdk_path (for incs and libs)
- superenv.rb add the right python include dir depending on
  whether a brewed python is installed or not.

Closes #15064.

Signed-off-by: Max Howell <mxcl@me.com>

Library/Formula/python.rb

@@ -21,40 +21,39 @@ class Distribute < Formula
 end
 
 class Pip < Formula
-  url 'http://pypi.python.org/packages/source/p/pip/pip-1.2.tar.gz'
-  sha1 '7876f943cfbb0bbb725c2761879de2889c1fe93b'
+  url 'http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz'
+  sha1 '35db84983ef3f66a8a161d320e61d192afc233d9'
 end
 
 class Python < Formula
-  homepage 'http://www.python.org/'
+  homepage 'http://www.python.org'
   url 'http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2'
   sha1 '842c4e2aff3f016feea3c6e992c7fa96e49c9aa0'
 
-  env :std
-
   depends_on TkCheck.new
   depends_on 'pkg-config' => :build
   depends_on 'readline' => :recommended
   depends_on 'sqlite' => :recommended
   depends_on 'gdbm' => :recommended
-  depends_on :x11 # tk.h includes X11/Xlib.h and X11/X.h
+  depends_on 'openssl' if build.include? 'with-brewed-openssl'
 
   option :universal
   option 'quicktest', 'Run `make quicktest` after the build'
+  option 'with-brewed-openssl', "Use Homebrew's openSSL instead of the one from OS X"
+  option 'with-poll', 'Enable select.poll, which is not fully implemented on OS X (http://bugs.python.org/issue5154)'
 
-  # --with-dtrace relies on CLT as the patch from
-  # http://bugs.python.org/issue13405 requires it.
-  # A note is added upstream about the CLT requirement.
+  # --with-dtrace relies on CLT as dtrace hard-codes paths to /usr
+  # http://bugs.python.org/issue13405
   option 'with-dtrace', 'Install with DTrace support' if MacOS::CLT.installed?
 
-  def site_packages_cellar
-    prefix/"Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages"
-  end
-
   def patches
     'https://raw.github.com/gist/3415636/2365dea8dc5415daa0148e98c394345e1191e4aa/pythondtrace-patch.diff'
   end if build.include? 'with-dtrace'
 
+  def site_packages_cellar
+    prefix/"Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages"
+  end
+
   # The HOMEBREW_PREFIX location of site-packages.
   def site_packages
     HOMEBREW_PREFIX/"lib/python2.7/site-packages"
@@ -65,11 +64,9 @@ def scripts_folder
     HOMEBREW_PREFIX/"share/python"
   end
 
-  def effective_lib
-    prefix/"Frameworks/Python.framework/Versions/2.7/lib"
-  end
-
   def install
+    opoo 'The given option --with-poll enables a somewhat broken poll() on OS X (http://bugs.python.org/issue5154).' if build.include? 'with-poll'
+
     # Unset these so that installing pip and distribute puts them where we want
     # and not into some other Python the user has installed.
     ENV['PYTHONPATH'] = nil
@@ -84,28 +81,56 @@ def install
            ]
 
     args << '--without-gcc' if ENV.compiler == :clang
+    args << '--with-pydebug' if ENV.compiler == :clang # http://bugs.python.org/issue13370
     args << '--with-dtrace' if build.include? 'with-dtrace'
 
-    # Further, Python scans all "-I" dirs but not "-isysroot", so we add
-    # the needed includes with "-I" here to avoid this err:
-    #     building dbm using ndbm
-    #     error: /usr/include/zlib.h: No such file or directory
-    ENV.append 'CPPFLAGS', "-I#{MacOS.sdk_path}/usr/include" unless MacOS::CLT.installed?
-
-    # Don't use optimizations other than "-Os" here, because Python's distutils
-    # remembers (hint: `python-config --cflags`) and reuses them for C
-    # extensions which can break software (such as scipy 0.11 fails when
-    # "-msse4" is present.)
-    ENV.minimal_optimization
-
-    # We need to enable warnings because the configure.in uses -Werror to detect
-    # "whether gcc supports ParseTuple" (https://github.com/mxcl/homebrew/issues/12194)
-    ENV.enable_warnings
-    if ENV.compiler == :clang
-      # http://docs.python.org/devguide/setup.html#id8 suggests to disable some Warnings.
-      ENV.append_to_cflags '-Wno-unused-value'
-      ENV.append_to_cflags '-Wno-empty-body'
-      ENV.append_to_cflags '-Qunused-arguments'
+    if superenv?
+      # To allow certain Python bindings to find brewed software:
+      cflags = "CFLAGS=-I#{HOMEBREW_PREFIX}/include"
+      ldflags = "LDFLAGS=-L#{HOMEBREW_PREFIX}/lib"
+      unless MacOS::CLT.installed?
+        # Help Python's build system (distribute/pip) to build things on Xcode-only systems
+        # The setup.py looks at "-isysroot" to get the sysroot (and not at --sysroot)
+        cflags += " -isysroot #{MacOS.sdk_path}"
+        ldflags += " -isysroot #{MacOS.sdk_path}"
+        # Same zlib.h-not-found-bug as in env :std (see below)
+        args << "CPPFLAGS=-I#{MacOS.sdk_path}/usr/include"
+      end
+      args << cflags
+      args << ldflags
+      # Avoid linking to libgcc http://code.activestate.com/lists/python-dev/112195/
+      args << "MACOSX_DEPLOYMENT_TARGET=#{MacOS.version}"
+      # We want our readline! This is just to outsmart the detection code,
+      # superenv handles that cc finds includes/libs!
+      inreplace "setup.py",
+                "do_readline = self.compiler.find_library_file(lib_dirs, 'readline')",
+                "do_readline = '#{HOMEBREW_PREFIX}/opt/readline/lib/libhistory.dylib'"
+
+    else
+      # This is obsolte with superenv (superenv defaults to -Os and does not
+      # disable warnings), but to support --env=std we leave it here:
+
+      # Python scans all "-I" dirs but not "-isysroot", so we add
+      # the needed includes with "-I" here to avoid this err:
+      #     building dbm using ndbm
+      #     error: /usr/include/zlib.h: No such file or directory
+      ENV.append 'CPPFLAGS', "-I#{MacOS.sdk_path}/usr/include" unless MacOS::CLT.installed?
+
+      # Don't use optimizations other than "-Os" here, because Python's distutils
+      # remembers (hint: `python-config --cflags`) and reuses them for C
+      # extensions which can break software (such as scipy 0.11 fails when
+      # "-msse4" is present.)
+      ENV.minimal_optimization
+
+      # We need to enable warnings because the configure.in uses -Werror to detect
+      # "whether gcc supports ParseTuple" (https://github.com/mxcl/homebrew/issues/12194)
+      ENV.enable_warnings
+      if ENV.compiler == :clang
+        # http://docs.python.org/devguide/setup.html#id8 suggests to disable some Warnings.
+        ENV.append_to_cflags '-Wno-unused-value'
+        ENV.append_to_cflags '-Wno-empty-body'
+        ENV.append_to_cflags '-Qunused-arguments'
+      end
     end
 
     if build.universal?
@@ -119,10 +144,11 @@ def install
     system "./configure", *args
 
     # HAVE_POLL is "broken" on OS X
-    # See: http://trac.macports.org/ticket/18376
-    inreplace 'pyconfig.h', /.*?(HAVE_POLL[_A-Z]*).*/, '#undef \1'
+    # See: http://trac.macports.org/ticket/18376 and http://bugs.python.org/issue5154
+    inreplace 'pyconfig.h', /.*?(HAVE_POLL[_A-Z]*).*/, '#undef \1' unless build.include? "with-poll"
 
     system "make"
+
     ENV.deparallelize # Installs must be serialized
     # Tell Python not to install into /Applications (default for framework builds)
     system "make", "install", "PYTHONAPPSDIR=#{prefix}"
@@ -141,41 +167,87 @@ def install
     # Symlink the prefix site-packages into the cellar.
     ln_s site_packages, site_packages_cellar
 
+    # Teach python not to use things from /System
+    # and teach it about the correct site-package dir because we moved it
+    sitecustomize = site_packages/"sitecustomize.py"
+    rm sitecustomize if File.exist? sitecustomize
+    sitecustomize.write <<-EOF.undent
+      # This file is created by `brew install python` and is executed on each
+      # python startup. Don't print from here, or else universe will collapse.
+      import sys
+
+      # Only do our fixes, if the currently run python is a brewed one.
+      if sys.executable.startswith("#{HOMEBREW_PREFIX}"):
+          # Fix 1)
+          #   A setuptools.pth and/or easy-install.pth sitting either in
+          #   /Library/Python/2.7/site-packages or in
+          #   ~/Library/Python/2.7/site-packages can inject the
+          #   /System's Python site-packages. People then report
+          #   "OSError: [Errno 13] Permission denied" because pip/easy_install
+          #   attempts to install into
+          #   /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
+          #   See: https://github.com/mxcl/homebrew/issues/14712
+          sys.path = [ p for p in sys.path if not p.startswith('/System') ]
+
+          # Fix 2)
+          #   pip install records installed files with relative paths and
+          #   because Homebrew installs python into #{HOMEBREW_PREFIX}/Cellar,
+          #   the symlinks for the executable python scripts is not correct, so
+          #   they are not uninstalled when `pip uninstall <x>` is called.
+          #   See: https://github.com/mxcl/homebrew/issues/14688
+          sys.prefix = "#{HOMEBREW_PREFIX}"
+    EOF
+
     # Tell distutils-based installers where to put scripts
-    scripts_folder.mkpath
-    (effective_lib+"python2.7/distutils/distutils.cfg").write <<-EOF.undent
+    (prefix/"Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/distutils.cfg").write <<-EOF.undent
       [install]
       install-scripts=#{scripts_folder}
     EOF
 
+    unless MacOS::CLT.installed?
+      makefile = prefix/'Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/Makefile'
+      inreplace makefile do |s|
+        s.gsub!(/^CC=.*$/, "CC=xcrun clang")
+        s.gsub!(/^CXX=.*$/, "CXX=xcrun clang++")
+        s.gsub!(/^AR=.*$/, "AR=xcrun ar")
+        s.gsub!(/^RANLIB=.*$/, "RANLIB=xcrun ranlib")
+      end
+    end
+
     # Install distribute and pip
-    Distribute.new.brew { system "#{bin}/python", "setup.py", "--no-user-cfg", "install", "--force", "--verbose" }
-    Pip.new.brew { system "#{bin}/python", "setup.py", "--no-user-cfg", "install", "--force", "--verbose" }
+    setup_args = ["-s", "setup.py", "--no-user-cfg", "install", "--force", "--verbose"]
+    Distribute.new.brew { system "#{bin}/python", *setup_args }
+    Pip.new.brew { system "#{bin}/python", *setup_args }
+
+    # It's important to have these installers in our bin, because some users
+    # forget to put #{script_folder} in PATH, then easy_install'ing
+    # into /Library/Python/X.Y/site-packages with /usr/bin/easy_install.
+    ['pip', 'pip-2.7', 'easy_install', 'easy_install-2.7'].each { |b| mv scripts_folder/b, bin }
   end
 
   def caveats
     <<-EOS.undent
-      The Python framework is located at
+      Homebrew's Python framework
         #{prefix}/Frameworks/Python.framework
 
-      You can find the Python demo at
+      Python demo
         #{HOMEBREW_PREFIX}/share/python/Extras
 
-      You can `brew linkapps` to symlink "Idle" and the "Python Launcher".
+      Distribute and Pip have been installed. To update them
+        pip install --upgrade distribute
+        pip install --upgrade pip
 
-      A "distutils.cfg" has been written, specifying the install-scripts folder as:
-        #{scripts_folder}
+      To symlink "Idle" and the "Python Launcher" to ~/Applications
+        `brew linkapps`
 
-      If you install Python packages via "pip install x" or "python setup.py install"
-      (or the outdated easy_install), any provided scripts will go into the
-      install-scripts folder above, so you may want to add it to your PATH.
+      You can install Python packages with (the outdated easy_install) or
+        `pip install <your_favorite_package>`
 
-      The site-package directory for brewed Python:
+      They will install into the site-package directory
         #{site_packages}
-
-      Distribute and Pip have been installed. To update them
-        #{scripts_folder}/pip install --upgrade distribute
-        #{scripts_folder}/pip install --upgrade pip
+      Executable python scripts will be put in:
+        #{scripts_folder}
+      so you may want to put "#{scripts_folder}" in your PATH, too.
 
       See: https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python
     EOS
@@ -185,8 +257,7 @@ def test
     # Check if sqlite is ok, because we build with --enable-loadable-sqlite-extensions
     # and it can occur that building sqlite silently fails if OSX's sqlite is used.
     system "#{bin}/python", "-c", "import sqlite3"
-    # See: https://github.com/mxcl/homebrew/pull/10487
-    # Fixed [upstream](http://bugs.python.org/issue11149), but still nice to have.
-    `#{bin}/python -c 'from decimal import Decimal; print Decimal(4) / Decimal(2)'`.chomp == '2'
+    # Check if some other modules import. Then the linked libs are working.
+    system "#{bin}/python", "-c", "import Tkinter"
   end
 end

Library/Homebrew/superenv.rb

@@ -137,8 +137,11 @@ def determine_cmake_include_path
     paths << "#{sdk}/usr/include/libxml2" unless deps.include? 'libxml2'
     if MacSystem.xcode43_without_clt?
       paths << "#{sdk}/usr/include/apache2"
-      # TODO prolly shouldn't always do this?
-      paths << "#{sdk}/System/Library/Frameworks/Python.framework/Versions/Current/include/python2.7"
+      paths << if Formula.factory('python').linked_keg.directory?
+        "#{HOMEBREW_PREFIX}/opt/python/Frameworks/Python.framework/Headers"
+      else
+        "#{sdk}/System/Library/Frameworks/Python.framework/Versions/Current/include/python2.7"
+      end
     end
     paths << "#{sdk}/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers/" unless x11?
     paths << "#{MacSystem.x11_prefix}/include" if x11?