psycopg2-binary: Why?

[jleclanche]

Forgive the blunt question but I cannot find discussion on why the split of psycopg2 and psycopg2-binary is happening. The current (2.7) way of doing wheel+source distribution works exactly how it's supposed to. Splitting into a separate -binary distribution actually breaks any library that depends on psycopg2=>2.x, which includes for example sqlalchemy's postgres bindings, because setuptools will complain the dependency is not matched.

[dvarrazzo]

The discussion is here

The explanation is here

If you have another solution let us know in the mailing list. The solution should involve not segfaulting.

[jleclanche]

Thanks for the context! This is super helpful.
I actually hit the issue specified there (uwsgi+psycopg2 libssl issues on debian). It's tough because I get that this stuff should work out of the box, but at the same time I think the current solution will only work out of the box as far as people won't be using psycopg2-binary.

If you are affected by that particular issue, compiling from source is one of the more appropriate fixes and pip has a command line flag for it (which psycopg2 itself documented in various places including this issue tracker).

I understand the desire to fix the issue (as I said, I'm affected by it), but I don't think this really fixes anything:

• Users who stay on psycopg2 now have a runtimewarning they may or may not understand
• pip install psycopg2 time and requirements changed with no clear explanation why without knowing the background or having seen aforementioned warning
• Users who switch to psycopg2-binary now have issues with various package requires such as the one I mentioned in this issue (sqlalchemy is a big user of psycopg2, this is probably going to surface a lot more once 2.8 is released)
• And for the users who switch to psycopg2-binary, the original issue with libssl still exists. It's just been moved.

I think the "solution" really is the --no-binary pip flag. Unless I completely misunderstand something, that has the same effect as moving the wheels to a different package, except that for this package the wheels will not have added issues.

[Cykooz]

I fully agree with @jleclanche. Creating of psycopg2-binary is not a solution for end users. On the contrary, it is a new problem for users whom uses many third-party libraries depended from psycopg2. Now one half of this libraries depends from psycopg2 (e.g. sqlalchemy), and other half - from psycopg2-binary (e.g. pgcli). Which version of psycopg2 package will be installed and used in that situation?

[Cykooz]

psycopg2-binary has not resolved problem with libssl.

Imagine, what if near future all python libraries will be depends from psycopg2-binary instead of psycopg2. What will this situation be different from when there was only one version of psycopg2?

[dvarrazzo]

@Cykooz use psycopg2 and forget the wheel for your use. Let the other decide for theirs. If they are writing multithread programs using ssh they better use psycopg2. Otherwise they are fine with -wheels. The libssl one is not a solvable problem. If it is a problem for you forget the wheels.

[Cykooz]

But not all depends from me. I use different third-party packages which already has psycopg2 or psycopg2-binary in self dependencies. And I can't control an order of installation of this packages. Sometimes psycopg2 will be installed first, other time - psycopg2-binary will.

By default pip don't use package name from METADATA to check that it already installed. It compare only package names taken from names of wheel files. This means that I can install both packages at the same time.

If I install psycopg2-binary first, and then install psycopg2:

$ pip install psycopg2-binary
$ pip install psycopg2

then if I uninstall psycopg2-binary:

$ pip uninstall psycopg2-binary

this will completely remove the code of psycopg2 from my python:

$ python
>>> import psycopg2
...
ImportError: No module named psycopg2

But pip will still consider that the psycopg2 is installed.

$ pip install psycopg2
Requirement already satisfied: psycopg2 in ./dist-packages

All of this is not very well.

[Cykooz]

Third-party packages should not determine which version of the psycopg2 distributive should be used. Because they do not know in what environment they will be used. Version of distributive of psycopg2 should be determined solely by external tools. For example, with help of the --no-binary option for pip.

[dvarrazzo]

@Cykooz What third party package depend on psycopg2-binary?

[Cykooz]

@dvarrazzo pgcli depends from psycopg2-binary.

[dvarrazzo]

@Cykooz pgcli is an interactive tool: it doesn't strike me as something that has to live 100% in the virtualenv of e.g. a Django webapp using psycopg. But those kind of tools are exactly the reason for which I don't want to get rid of the wheels altogether. It's probably worth a chat with them to understand what's the best strategy.

[Cykooz]

Oh, maybe you are in something right.

I'm just afraid that not only standalone applications will include psycopg2-binary as a dependency, but also libraries.

[dvarrazzo]

@Cykooz I appreciate your concern. And it highlights the fact that if some library started depending on that package it may cause interdependencies problem: it's something to highlight very clearly in the docs. We are a bit into uncharted waters here: we are trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it.