OS X installer packages should be signed for OS X 10.8 Gatekeeper feature

[ned-deily]

BPO	15661
Nosy	@malemburg, @jcea, @ronaldoussoren, @ned-deily, @florentx, @rovitotv
Files	pkgbuild.plist

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2016-07-23.18:42:16.200>
created_at = <Date 2012-08-15.06:55:07.246>
labels = ['OS-mac', 'type-bug', 'expert-installation', 'build']
title = 'OS X installer packages should be signed for OS X 10.8 Gatekeeper feature'
updated_at = <Date 2016-07-23.18:42:16.199>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2016-07-23.18:42:16.199>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2016-07-23.18:42:16.200>
closer = 'ned.deily'
components = ['Build', 'Installation', 'macOS']
creation = <Date 2012-08-15.06:55:07.246>
creator = 'ned.deily'
dependencies = []
files = ['30827']
hgrepos = []
issue_num = 15661
keywords = []
message_count = 9.0
messages = ['168262', '168263', '192459', '192462', '192515', '192640', '227143', '227154', '271068']
nosy_count = 7.0
nosy_names = ['lemburg', 'jcea', 'ronaldoussoren', 'ned.deily', 'flox', 'Todd.Rovito', 'python-dev']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15661'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[ned-deily]

OS 10.8 Mountain Lion includes a new feature that "helps protect users from downloading and installing malicious software" by providing a mechanism for developers to sign their installable objects with a unique Apple-signed Developer ID certificate. The default security policy for 10.8 is to only allow installation of items that have been signed. The user is able to override that policy either persistently for all installs (using System Preferences) or selectively for individual unsigned items (for example, by using control-click and "Open with").

OS X allows various items to be signed, including executables, application bundles, application frameworks, and installer packages. The python.org OS X installations includes all of these. For applications to be made available through the Mac App Store, all of these have to be signed and there are additional requirements, such as sandboxing and restrictions on API usage, that are not compatible with a general-purpose programming environment such as Python provides. Therefore, this issue is only concerned with meeting the requirements for 10.8 Gatekeeper and "Distributing Outside the Mac App Store" (see references below).

Each python.org OS X release is delivered via an OS X installer package (.mpkg) within an OS X diskimage file (.dmg) and installed using the standard OS X installer. So, at a minimum, the only entity that needs to be signed is each installer package we produce. Unfortunately, the Python installer build process (as encapsulated in Mac/BuildScript/build-installer.py) currently packages releases in an old deprecated bundle (non-flat) metapackage format that cannot be signed. That means build-installer.py and the manual process surrounding releases need to be updated to produce the newer flat package formats that can then be signed using a unique Developer ID Installer certificate requested from and signed by Apple.

Several points:

1. The changes needed to support the newer sign-able installer formats should be able to be limited to python-installer.py itself and so should minimize risk to other releases and Python itself. Likewise, to the end user, the installation process should look (nearly) identical to the current process as the same system installer app is used. Of course, the install process will need to be tested to ensure that the new format packages produce the same results (i.e. files, permissions) on the user system as the old format packages do. It may be possible and desirable to include these changes in the upcoming 3.3.0 release or, if not available in time, in a subsequent 3.3.x maintenance release. (I am currently working on the changes.)

2. There is a process question of what Developer ID(s) to use for requesting Installer certificates for singing. As documented, Apple requires the requestor of a singing certificate to be a member of the Mac Developer Program which normally involves a modest annual fee. One can be a member of the developer program as an individual developer or as a member of a development team associated with a company or other legal entity. This distinction affects how the installation is presented to the user. I believe that we should aim to have a Python Software Foundation development team membership with the builders of python.org releases as team members as needed (currently that means me with Ronald as backup). It is beyond the scope of this issue to define and implement that process. In the immediate future, once the new installer package is supported, in lieu of a PSF-backed certificate, it *may* be desirable to sign the package with an individual certificate (both Ronald and I are individual members of the developer program). That is roughly analogous to the current practice of using individual release team members' PGP keys to sign Python release artifacts (source tar balls and binary installers) that are downloadable from python.org although it may not be as visible to the user as the Gatekeeper signing. We will pursue the development team option outside of this tracker issue with PSF officers.

3. I believe it is the case that the newer flat package formats are not supported on OS X 10.3 and only partially supported on OS X 10.4. For Python 2.7.x and 3.2.x, we have been producing two installers: a 32-bit-only installer for 10.3+ and a 64-bit/32-bit installer for 10.6+. For Python 3.3, we have changed the minimum requirements for the 32-bit-only installer to be 10.5+. So it should be possible to move all 3.3.x installers to the new format and sign them. For older releases, we will have to look at the tradeoffs. Since we have kept the build-installer.py script identical for current releases of Python 2.7.x and Python 3.2.x, additional implementation costs and risks are small: the modified 3.3.0 script should work for 3.2.x and 2.7.x releases as well. Without introducing ABI incompatibilities within a major release cycle, one option may be to: do not change the 2.7.x 32-bit-installer format (so that it is still usable on 10.3 and 10.4), change the 2.7.x 64-bit-installer to the new format so it can be signed, and either do the same for 3.2.x or do nothing new (since 3.2.x will be entering security-fix-only mode sometime after the release of 3.3.0).

The open issues should be decided after implementing the new format support and testing to see if the assumptions are correct.

References:

http://support.apple.com/kb/HT5290
https://developer.apple.com/resources/developer-id/
https://developer.apple.com/programs/mac/team.html

[ned-deily]

An additional point: the 3.3.0 Installer README and python.org web pages need to be updated to incorporate 10.8 installation information as necessary prior to final release. (in progress)

Also, s/singing/signing/

[ronaldoussoren]

Creating flat packages should be easy enough, I'm hoping to experiment with them during the sprints at EP (but we're already at the end of the first day). At first I feared that creating installers in the new format would require using GUI tool, but it seems that the creating can be scripted using command line tools (like pkgbuild).

According to <http://s.sudre.free.fr/Stuff/Ivanhoe/FLAT.html\> flat packages were introduced in OSX 10.5, which would mean we could only move the "intel" installer to that format.

It would IMHO be worthwhile to do that for all new stable releases (2.7.x, 3.3.x and 3.4.x), while at the same time further reducing the visibility of the 32-bit installers (as those should only be used by users that just cannot use the newer installers).

As to the question of actually signing the installer: we ask the PSF about acquiring a developer account once we have something that can actually be signed.

[ronaldoussoren]

I just created a very crude package with the "new" tools.

$ pkgbuild --analyze --root /tmp/_py/_root pkgbuild.plist
...

$ pkgbuild --root /tmp/_py/_root --identifier org.python.python34 --component-plist pkgbuild.plist pythoninstall.pkg
pkgbuild: Reading components from pkgbuild.plist
pkgbuild: Adding component at Library/Frameworks/Python.framework/Versions/3.4
pkgbuild: Adding component at Applications/Python 3.4/IDLE.app
pkgbuild: Adding component at Applications/Python 3.4/Python Launcher.app
pkgbuild: Adding component at Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app
 pkgbuild: Wrote package to pythoninstall.pkg

That package can then be opened using installer.app, but isn't usable as it.

!) There are no pre/post scripts

2. User cannot select components

3. There is no README, License, custom logo, ...

4. I'm pretty sure this doesn't install the documentation

It might be possible to add all missing features using pkgbuild and productbuild, but I'm not sure about that.

[ronaldoussoren]

Also worthwhile to look into: http://s.sudre.free.fr/Software/Packages/about.html

This is a GUI tool for creating packages, with a command-line tool for scripting. At the very least we could use this to check if it is possible to build a flat installer that does what we want.

[malemburg]

On 06.07.2013 18:17, Ronald Oussoren wrote:

As to the question of actually signing the installer: we ask the PSF about acquiring a developer account once we have something that can actually be signed.

Please write to psf@python.org with an explanation of how this can be
done. We'd also need to know who will be in charge of the signing
(probably the core dev doing the respective release).

[python-dev]

New changeset 569a889e3b6c by Ned Deily in branch '3.4':
Issue bpo-15661: Update OS X installer welcome and readme files for 3.4.2.
https://hg.python.org/cpython/rev/569a889e3b6c

[ned-deily]

After a fair amount of research, trial-and-error, and testing on all of the OS X platforms we support, I think we're now ready to switch to signed flat packages for the OS X installers and thereby greatly improve the user installation experience on current systems. I've found that it *is* possible to produce signed installers that work as expected on the same set of o/s releases and platforms as the current legacy installers (i386/ppc for 10.5+ and x86_64/i386 for 10.6+), with the same set of user installation options, although there were a number of gotchas along the way. One unexpected issue was that our Python frameworks are not immutable - since they include install paths such as the site-packages and scripts directories - and the current Apple packaging tools aren't designed to support a configuration like that. On newer releases where codesigning is fully supported, the installer wants to treat framework bundles as atomic elements which, I discovered, can result in the user's site-installed packages being deleted during an maintenance release upgrade. I was able to work around that in a way that works across all supported releases but it does point out that we should consider modifying our framework deployments to move to "read-only" bundles if possible.

We produce the two installer variants by running build-installer.py on two different OS X systems (10.5 and 10.6) and then move the resultant bundle installers encapsulated in OS X disk image files (.dmg) to a third current (10.9) system for pgp signing, archiving, and uploading. The new process, beginning with the upcoming 3.4.2, remains the same except that the build artifacts from the two builds are now post-processed on the 10.9 system using the current Apple packaging tools to produce the flat package installer files and to do the "Gatekeeper" signings. For 3.4.2rc1, the configuration files for the packaging tools have been manually generated. The next step, probably post-3.4.2, will be for build-installer.py to auto-generate the config files.

Another advantage of flat-format installers is that they are a compressed single file and don't need to be wrapped in another container like the legacy bundle installers do. So, starting with 3.4.2, the installer file (.pkg) will be downloaded directly rather than within a .dmg file. This simplifies the user installation process: double-clicking on the downloaded .pkg file will start the installer directly rather than having to click on the downloaded dmg, waiting for it to mount, double-clicking on the installer file in the window that opens, and remembering to dismount the volume when the install is complete. And, in some browser configurations, the installer will be launched automatically, although in all cases, user action is still required to complete the install. One possible downside to eliminating the dmg (or similar) container is that there will no longer be a redundant readme file outside of the installer. The installer presents the package's welcome, readme, and license files and the opportunity for the user to save and/or print each, but previously we also made a copy of the readme file available in the dmg directory next to the installer file. The primary case where that could be useful is if the user cannot launch the installer for some reason and the readme provides a workaround. But that unusual case is the one we are fixing by moving to signed packages. Not surprisingly, many users do not read readmes anyway: the previous readme files did include instructions on how to workaround the Gatekeeper issue yet many users still had problems. In the unlikely event that a similar problem arises, we still have the option to revert to a zip or a dmg if just providing info on the python.org download pages proves insufficient.

To address the points in the original issue: (1) I think the risk has been minimized as there are no changes to the actual installer builds and the user experience in the installer is virtually unchanged; what we have changed is removing obstacles to getting to the installer. (2) While I still think in the long run it would be best to have a PSF-owned Apple developer company account for signing (and I still intend to pursue that with the board at some point after coming up with a concrete proposal), I don't think it is a particularly pressing issue. Initially, the installers are being signed with my personal developer id, just as other release files are signed by the personal pgp keys of the release team members who produce them. Unlike the Windows installer, the OS X installer does not display information about the signing key unsolicited other than an indication that one is present if you know where to look. There's a small icon indicating a signed installer in the upper-right corner of the installer window; clicking it causes the certificate info to be displayed. (3) is no longer an issue. As noted, we have already dropped installer support for OS X 10.3 and 10.4 in 3.x releases. Because of the extended life of the 2.7 series, with 2.7.7, we started the deprecation of those old systems by adding a 10.5+ format installer similar to 3.x releases while continuing to provide a 10.3+ format. With the next 2.x release (2.7.9) we plan to drop the 10.3+ installer altogether. So, going forward, all release installers will be flat and signed. Note that the daily dmg buildbot is still producing (old-format) test installers on a 10.4 system, which should help to ensure we don't needlessly break things on older systems.

[ronaldoussoren]

Ned,

Can this issue be closed? The current installers on www.python.org are signed.

Ronald