Distribute Python gRPC via PyPI

[nathanielmanistaatgoogle]

Simple installation is now a priority.

[nathanielmanistaatgoogle]

This afternoon I had a chance to talk with @jgeewax about this; he strongly encouraged us to drop the grpc-c-core-must-be-installed-on-the-system requirement if we can. Mostly his guidance was "be packaged as conveniently as pycrypto", which is one-line-installable with pip and is also separately installable as an Ubuntu package ("apt-get install python-crypto").

So that's a potential objective.

[nathanielmanistaatgoogle]

#1154 got a start on this and depending on how one scopes this issue may have completed it.

Let's wait a week or so and then ask ourselves what remains here? The big choice is whether or not to include the whole library in the package so that the "install the core library first" requirement is dropped, right?

[jgeewax]

Pushing out deb files is awesome progress -- thanks for the link to that! I still think it'd be awesome if this was an apt-get/yum/whatever else for the main linux distros, and brew/port for Mac, and a proper msi for Windows.

For Python (and I assume others) it'd be really nice if the whole install for this was pip install grpc. Without that, a library that depends on this now has some underlying dependencies that users have to figure out on their own. This means that pip install my-library (depending on gRPC) would fail because the gRPC core package isn't there -- which makes me sad :(

So in summary:

• apt package (sudo apt-get install grpc)
• yum package (sudo yum install grpc)
• homebrew package (sudo brew install grpc)
• macports package (sudo port install grpc)
• Windows installer (::double click:: a .msi file, Next, Next, Install, Done)
• PyPI package contains everything needed (pip install grpc gives me everything I need)

I totally get that this is a crap-ton of maintenance... I hope that gRPC settles enough to make it worthwhile...

(BTW, please tell me if this it the wrong place to make these requests....)

[nathanielmanistaatgoogle]

This issue tracker is the right place, but this particular issue is more narrowly scoped than what you've just enumerated. Let's open separate issues for each separable task you describe.

apt: #1208
yum: #1209
homebrew: #1210
macports: #1211
Windows: #1212

This issue remains open specifically for the question of our distribution on PyPI.

[soltanmm]

So is the scope of this issue to distribute a package via PyPI or to distribute the whole of Python GRPC s.t. users need not install the core libraries separately?

[nathanielmanistaatgoogle]

@soltanmm: that's to-be-decided... in this issue. While I know that one-step install is massively convenient I haven't yet started to work up an estimate of the effort involved.

[soltanmm]

After much discussion, we've come to the (tentative) conclusion that the trade-offs associated with packaging GRPC Python via PIP such that everything comes installed in one go is infeasible or undesirable.

In particular, we hit two issues: one of the quagmire that is OpenSSL, and (to a lesser degree) that of the culture surrounding package management on, in particular, Linux.

Why is OpenSSL a problem?

For reasons not entirely clear, there's an aversion to packaging OpenSSL as a binary, and building OpenSSL from source is notoriously complicated to begin with, let alone in the manner appropriate for GRPC. The binary issue stems from discomfort with respect to the U.S.'s stance on cryptography as weapons and thus not things we should inadvertently be responsible for exporting from the country over the network. It's unclear to us developers (we've yet to consult with our lawyer-cats) whether or not this issue propagates to the source code.

We furthermore have the issue that the version of OpenSSL on which we depend (≥1.0.2a) is not widely available on systems by default.

Despite GRPC core being buildable without SSL support, it's in our interest to make being secure as easy as possible. We don't want to end up in situations e.g. where if the user installed GRPC, got an insecure version, and later wanted to go secure, then the user'd need to jump through hoops to get the secure version.

Why is packaging a problem?

In the Linux culture, it's desirable to have distinct packageable items be in distinct packages. Shared objects get their own version numbers, one links to specific libraries and their versions, etc. Under Debian, for example, we'd eventually want some manner of python-grpc package to directly depend on an e.g. libgrpc package (and a libgpr package?) and thus to indirectly depend on OpenSSL ≥1.0.2a. PyPI only supports package dependencies among Python packages (it is, after all, a Python repository). This entails that we cannot (sanely) separate out GRPC Python from GRPC core on PyPI alone, while we can separate them if we assume the user is willing to install GRPC core separately.

PySqlite3, for example, took this approach back before it became part of Python's 'standard' distribution. Unless sqlite3.c was somehow provided to the setup script (which may have been the case in a much earlier incarnation of the library, but not so in the latest version), the package would depend on the system install (and presumably fail to build if the system lacked the library).

Then there's OpenSSL...

pyOpenSSL employs FFI to communicate with OpenSSL (thus implying its implicit requirement that OpenSSL be installed on the system first). eGenix did back-breaking setup.py acrobatics with their pyOpenSSL-like package to ensure that OpenSSL would be installed on the system (and that for a version incompatible with GRPC core's requirements).

So, even if we packaged GRPC core in with GRPC Python, we'd still be in the same boat with ensuring OpenSSL's availability (and that appears to currently require more engineering effort than we are willing to apply).

What are some potential solutions we're thinking about?

(This section is by no means exhaustive - I'm typing this on an empty stomach and desperately need lunch. :-P )

Remember that GRPC needs to communicate with the binary of OpenSSL. It's conceivable that if eGenix's pyOpenSSL package upgraded from 1.0.1m to 1.0.2a we'd be able to import their Python package and have the linker handle the GRPC core library dependencies at link time (thus allowing us to use OpenSSL). That would allow us to, perhaps, package GRPC core in with GRPC Python on PyPI, and not require any package management outside pip. Such is an option we're keeping an eye on.

We also have the option of dynamically choosing whether or not to use the secure version of GRPC at setup time. We could query the system for OpenSSL, and if it isn't there use libgrpc_unsecure. This goes against our desire to make being secure as easy as possible s.t. users who installed GRPC and later installed OpenSSL would need to reinstall GRPC to have security enabled.

There's a security credentials extension API in the works that could potentially solve this issue by having an OpenSSL module-of-sorts for security. There are some problems with that theoretical future (in particular making it enabled by default wherever possible), but it may be possible.

I'm closing this issue (for now). I'd be happy to have it re-opened it with new information or resources.

[ppg]

@soltanmm perhaps the solution to "For reasons not entirely clear, there's an aversion to packaging OpenSSL as a binary ..." is to build two packages, one with openssl embedded and one without; then if people have an aversion they can use the one without and figure out how to get 1.0.2 onto their distro; if not they can use the embedded one and call it a day.

[soltanmm]

@ppg I don't believe we're worried about users' aversions to a packaged OpenSSL. We're currently worried about possible import/export restrictions surrounding cryptographic software (among other possible legal constraints); the concerns extend beyond Python, and are murky. I believe the team will be checking into that to clear things up soonish.