psycopg2 dependency, either source or binary

[j-bennet]

Dear psycopg2 developers,

I work on pgcli, a console application for Postgres written in Python. Ever since psycopg2 was split into binary and source packages, we've been getting issues from people who want to install a binary distribution of psycopg2 with pgcli (because building from source either breaks, or takes too long for them, etc), and can't figure out a good way to do it. Example:

dbcli/pgcli#1085
dbcli/pgcli#1131

We have psycopg2 in our install_requires:

https://github.com/dbcli/pgcli/blob/e0a4c18c4a074016beef46886b03e456102e2db1/setup.py#L15

Pgcli can't run without either psycopg2 or psycopg2-binary being installed. We don't care which one, either one is fine, but it's not an optional dependency and we can't move it to extras_require.

Before 2.8, we gave users who have problems installing psycopg2 from source to force binary install like this:

pip install pgcli --only-binary psycopg2

Since 2.8, this of course does not work anymore, because the psycopg2 package is source-only:

ERROR: Could not find a version that satisfies the requirement psycopg2>=2.8 (from pgcli) (from versions: none)
ERROR: No matching distribution found for psycopg2>=2.8 (from pgcli)

With setuptools, it does not seem that there's any way to specify depencency on "either psycopg2 or psycopg2-binary". We considered:

1. Removing psycopg2 from install_requires and using a post-install script to install it. Not great, because this is not going to work with wheels.
2. Switching to psycopg2-binary in install_requires. We did this briefly, but had to revert because someone had a problem with the binary package.
3. Making both psycopg2 and psycopg2-binary optional packages. Not great either, because this needs extra steps from users to install pgcli completely and get a working application.

I don't know all the reasoning behind splitting psycopg2 into 2 different packages, but I really wish psycopg2 remained a single package with an option for users to use --no-binary and --only-binary, and without requiring workarounds.

I hope you can advise, having the insider view of psycopg2. What is the right thing to do for pgcli, in your opinion? What can we advise to users that:

• want to use psycopg2
• want to use psycopg2-binary

and how can we ensure that pgcli is successfully installable for both?

Thank you!

Irina Truong
pgcli maintainer

[dvarrazzo]

Hello Irina,

Unfortunately we had to drop the binary wheel from the psycopg2 package for problems of unreliability that can be easily found back searching in the github issues. Pip/python packages don't allow for a "prefer source" choice, so we had to make our "opt-in mechanism", and I agree it isn't pretty.

The only thing we suggested for your project is to depend on psycopg2. The -binary package is not available for all the platform: binary wheels are not available for Alpine Linux because there is no standard tagging, for instance, it's not possible to build wheels on Apple M1, and anyway in general Apple will break binary compatibility every 6 months. You don't want to depend on the -binary package.

It would be nice if it could be declared that both psycopg2 and psycopg2-binary package "provide psycopg2", there has been some discussion on the packaging groups but I am not aware of any progress.

Short of dropping the -binary package altogether, given the limitation of the python packages metadata, I don't think we can do anything helpful.

I have a proposal: what about switching pgcli to Psycopg 3? The advantages are:

• There is a pure python module to depend on, psycopg. If people can and want to install from binary they can use an extra and install psycopg[binary]. See https://www.psycopg.org/psycopg3/docs/basic/install.html for details.
• You can help us to debug it, and we can understand, before it's too late, if the design choices we made make sense (in the packaging area and not).
• On the other hand, if you need help with your project, I can give you some.
• Strictly typed, asynchronous, and many other features which you might need or not, but if you want they are there :)

FYI I have just finished adapting Django to psycopg3 and I have written FastAPI applications using it, so I believe stuff more complicated than what pgcli does have already been dealt with.

[j-bennet]

pip/python packages don't allow for a "prefer source" choice

There's a --no-binary option. Or do you mean that by default, binary is preferred if available?

[dvarrazzo]

That's correct: if there are binary packages available, they are installed by default.