Include OpenSSL in our Windows wheels

[alex]

Installing OpenSSL isn't super easy, even with the binaries for it. Maybe we should bundle it?

[alex]

A big downside is that this means any release for upstream requires a release for us today (including, and especially, security releases), and users will need to be trained to upgrade us. I'm not sure that something like heartbleed would necessarily trigger in people's minds that thye need to upgrade PyCA cryptography.

[reaperhulk]

This definitely would dramatically ease the use of cryptography on Windows. If we do release a new wheel when OpenSSL is updated what versioning do we use? sdist users would get no code changes so do they just recompile for no reason?

[dreid]

@alex says just bump the version and re-release everything for everyone.

[alex]

If you've already compiled PyCA cryptography once, doing it again isn't a
particular burden.

On Mon, Jun 9, 2014 at 4:28 PM, David Reid notifications@github.com wrote:

@alex https://github.com/Alex says just bump the version and re-release
everything for everyone.

—
Reply to this email directly or view it on GitHub
#1121 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[public]

I agree with @alex

[reaperhulk]

Okay, well I'm 👍 on doing this mostly because I feel terrible every time I have to explain to a Windows user how to install this project. If there's consensus we could change to static wheels on Windows for 0.5.

[dstufft]

Techincally wheels have a build number, although I'm not sure if it's exposed in an easy to use fashion or anything. I'd have to double check to make sure that it worked how I'm remembering and test it to make sure it actually worked, but it's theoretically there and can be used.

Course just jumping the version isn't a big deal either.

[dstufft]

Something that @Lukasa brought up. If we're bundling OpenSSL for Windows should we also do it for OSX? Even though OSX has a copy of OpenSSL it's an unmaintained copy.

[reaperhulk]

Well, not unmaintained, but they'll never upgrade from 0.9.8 (and it will become unmaintained if the OpenSSL developers cut off 0.9.8 support).

If we can target major OS X releases reliably I guess I'd support doing 10.7 through 10.9 wheels including OpenSSL statically (assuming we choose to do so for Windows).

[dstufft]

If by "target" you mean "Will pip select the correct Wheel", then the answer is yes. In 1.4.x and 1.5.x pip does exact matches against the OSX version in the filename (which I believe is taken from the SDK version that was compiled against). In pip 1.6.x that changes so that it does >= on the OSX version in the wheel filename (again the SDK version it was compiled against) since according to OSX people that's a safe thing to do.

[reaperhulk]

pyenv's pythons are compiled against the 10.4 SDK (pip wheel will provide macosx_10_4_x86_64), so does that means we'd need to custom compile all supported pythons on each supported OS X release and then build wheels from those?

[dstufft]

Do you want a different wheel for every OSX release, or is one wheel for all OSX releases 10.4+ OK.

[reaperhulk]

CommonCrypto forces us to require a different wheel for each release.

[dstufft]

Unless distutils provides a way to specify the SDK you want to build against (I don't know if it does, AFAIK it just uses whatever SDK CPython was built against) then yes you'd need to compile that. Yet another reason to split the backends out!

But CommonCrypto really doesn't have backwards compatibility like the SDK does?

[reaperhulk]

The commoncrypto API expanded substantially from 10.7 to 10.8 (but did not change in 10.9).  Since we need to bind to symbols available only in 10.8+ we would also need a 10.7 wheel that doesn't include commoncrypto.

On Sat, Jun 14, 2014 at 4:02 PM, Donald Stufft notifications@github.com
wrote:

Unless distutils provides a way to specify the SDK you want to build against (I don't know if it does, AFAIK it just uses whatever SDK CPython was built against) then yes you'd need to compile that. Yet another reason to split the backends out!

But CommonCrypto really doesn't have backwards compatibility like the SDK does?

Reply to this email directly or view it on GitHub:
#1121 (comment)

[dstufft]

Ok, so we'd need a macosx_10_8_x86_64 wheel and a macosx_10_4_x86_64 (or whatever our minimum version is without CC) wheel.

[reaperhulk]

That will only work with pip 1.6 (which isn't released yet?) correct?

[dstufft]

Depends on what you mean by work. On pip 1.4 and 1.5 it will be as if you do not have Wheels for anything but 10.8 and 10.4. Which is already the case since as of right now we're not publishing Wheels for OSX at all. On pip 1.6+ it will be as if you published Wheels for everything 10.4+.

[Ayrx]

I am +1 on static builds for Windows wheels.

For OS X though, I'm not so sure. What algorithms is missing from commoncrypto that requires OpenSSL? Ideally, I think not requiring OpenSSL at all on OS X by default is best. This would involve splitting up the backends that should hopefully be the focus of 0.6 though.

[reaperhulk]

The RSA/DSA/EC support in Security.framework is not (nearly) as flexible as what is in OpenSSL. There are some definite advantages to using OpenSSL in certain cases on OS X. But I am fine with doing windows static builds now and revisiting the (orthogonal) OS X wheel issue later.

[Ayrx]

Sounds good to me.

[reaperhulk]

Okay, if we're going to move forward on this here are the steps:

• Change jenkins jobs to add the lib\VC\static directories to the LIB variable when building. (master, PR, and wheel jobs)
• Submit PR that alters the libraries we link against on Windows (libeay32 becomes libeay32MT, etc).
• Update windows documentation to explain static compile steps.

Does anybody have an objection to doing this for 0.5?

[Ayrx]

Have we settled on how to do versioning yet? Is the build number thing @dstufft mentioned doable or do we have to increment the version for any OpenSSL updates?

Either way, I'm 👍 on doing this for 0.5.

[alex]

One note: since we'll have to do releases for upstream OpenSSL security releases, we should figure out what the appropriate channels for publication of those are. Initial thoughts:

• Our own mailing list
• Twisted mailing list
• pyOpenSSL mailing list
• requires.io security list

[public]

Have we figured out the packaging for this yet? Bumping the version number seems like it would work but would have an effect on all our platforms without actually fixing anything on platforms we don't ship OpenSSL on. Seems like that might be a bit confusing to users? If we get our release wording right I suppose that might be good enough though.

[Ayrx]

Like @dstufft mentioned, it appears as though the wheels format has a build number. PEP 0427, File Contents, Point #11. Is this something that we can set from setuptools?

@dstufft needs to weigh in here, he's the expert in all things packaging after all. :)

[reaperhulk]

Despite it affecting all platforms I prefer incrementing the primary version. Otherwise when talking about security releases we'll need to do things like say "0.5 wheel build 1 is vulnerable" which is terrible. I'm not even sure how you easily see what wheel version was installed (pip list does not appear to show it).

[public]

I agree with @reaperhulk. I was thinking maybe that platform specific
dependencies might be a thing so we could put OpenSSL in its own package
but that's probably overkill at least until we have figured out how to
separate the backends from the main package.

On 28 June 2014 15:16:24 Paul Kehrer notifications@github.com wrote:

Despite it affecting all platforms I prefer incrementing the primary
version. Otherwise when talking about security releases we'll need to do
things like say "0.5 wheel build 1 is vulnerable" which is terrible. I'm
not even sure how you easily see what wheel version was installed (pip list does not appear to show it).

---

Reply to this email directly or view it on GitHub:
#1121 (comment)

[alex]

I think we should just bump the patch level (i.e. 0.5.{HERE}) and be done
with it.

On Sat, Jun 28, 2014 at 7:49 AM, Alex Stapleton notifications@github.com
wrote:

I agree with @reaperhulk. I was thinking maybe that platform specific
dependencies might be a thing so we could put OpenSSL in its own package
but that's probably overkill at least until we have figured out how to
separate the backends from the main package.

On 28 June 2014 15:16:24 Paul Kehrer notifications@github.com wrote:

Despite it affecting all platforms I prefer incrementing the primary
version. Otherwise when talking about security releases we'll need to do
things like say "0.5 wheel build 1 is vulnerable" which is terrible. I'm
not even sure how you easily see what wheel version was installed (pip list does not appear to show it).

---

Reply to this email directly or view it on GitHub:
#1121 (comment)

—
Reply to this email directly or view it on GitHub
#1121 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084