Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

psycopg2-binary: Why? #674

Closed
jleclanche opened this issue Feb 11, 2018 · 33 comments
Closed

psycopg2-binary: Why? #674

jleclanche opened this issue Feb 11, 2018 · 33 comments
Labels
wheel Bugs related to the wheel distribution, not happening installing from source

Comments

@jleclanche
Copy link

Forgive the blunt question but I cannot find discussion on why the split of psycopg2 and psycopg2-binary is happening. The current (2.7) way of doing wheel+source distribution works exactly how it's supposed to. Splitting into a separate -binary distribution actually breaks any library that depends on psycopg2=>2.x, which includes for example sqlalchemy's postgres bindings, because setuptools will complain the dependency is not matched.

@dvarrazzo
Copy link
Member

The discussion is here

The explanation is here

If you have another solution let us know in the mailing list. The solution should involve not segfaulting.

@jleclanche
Copy link
Author

Thanks for the context! This is super helpful.
I actually hit the issue specified there (uwsgi+psycopg2 libssl issues on debian). It's tough because I get that this stuff should work out of the box, but at the same time I think the current solution will only work out of the box as far as people won't be using psycopg2-binary.

If you are affected by that particular issue, compiling from source is one of the more appropriate fixes and pip has a command line flag for it (which psycopg2 itself documented in various places including this issue tracker).

I understand the desire to fix the issue (as I said, I'm affected by it), but I don't think this really fixes anything:

  • Users who stay on psycopg2 now have a runtimewarning they may or may not understand
  • pip install psycopg2 time and requirements changed with no clear explanation why without knowing the background or having seen aforementioned warning
  • Users who switch to psycopg2-binary now have issues with various package requires such as the one I mentioned in this issue (sqlalchemy is a big user of psycopg2, this is probably going to surface a lot more once 2.8 is released)
  • And for the users who switch to psycopg2-binary, the original issue with libssl still exists. It's just been moved.

I think the "solution" really is the --no-binary pip flag. Unless I completely misunderstand something, that has the same effect as moving the wheels to a different package, except that for this package the wheels will not have added issues.

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

I fully agree with @jleclanche. Creating of psycopg2-binary is not a solution for end users. On the contrary, it is a new problem for users whom uses many third-party libraries depended from psycopg2. Now one half of this libraries depends from psycopg2 (e.g. sqlalchemy), and other half - from psycopg2-binary (e.g. pgcli). Which version of psycopg2 package will be installed and used in that situation?

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

psycopg2-binary has not resolved problem with libssl.

Imagine, what if near future all python libraries will be depends from psycopg2-binary instead of psycopg2. What will this situation be different from when there was only one version of psycopg2?

@dvarrazzo
Copy link
Member

@Cykooz use psycopg2 and forget the wheel for your use. Let the other decide for theirs. If they are writing multithread programs using ssh they better use psycopg2. Otherwise they are fine with -wheels. The libssl one is not a solvable problem. If it is a problem for you forget the wheels.

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

But not all depends from me. I use different third-party packages which already has psycopg2 or psycopg2-binary in self dependencies. And I can't control an order of installation of this packages. Sometimes psycopg2 will be installed first, other time - psycopg2-binary will.

By default pip don't use package name from METADATA to check that it already installed. It compare only package names taken from names of wheel files. This means that I can install both packages at the same time.

If I install psycopg2-binary first, and then install psycopg2:

$ pip install psycopg2-binary
$ pip install psycopg2

then if I uninstall psycopg2-binary:

$ pip uninstall psycopg2-binary

this will completely remove the code of psycopg2 from my python:

$ python
>>> import psycopg2
...
ImportError: No module named psycopg2

But pip will still consider that the psycopg2 is installed.

$ pip install psycopg2
Requirement already satisfied: psycopg2 in ./dist-packages

All of this is not very well.

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

Third-party packages should not determine which version of the psycopg2 distributive should be used. Because they do not know in what environment they will be used. Version of distributive of psycopg2 should be determined solely by external tools. For example, with help of the --no-binary option for pip.

@dvarrazzo
Copy link
Member

@Cykooz What third party package depend on psycopg2-binary?

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

@dvarrazzo pgcli depends from psycopg2-binary.

@dvarrazzo
Copy link
Member

@Cykooz pgcli is an interactive tool: it doesn't strike me as something that has to live 100% in the virtualenv of e.g. a Django webapp using psycopg. But those kind of tools are exactly the reason for which I don't want to get rid of the wheels altogether. It's probably worth a chat with them to understand what's the best strategy.

@Cykooz
Copy link

Cykooz commented Mar 7, 2018

Oh, maybe you are in something right.

I'm just afraid that not only standalone applications will include psycopg2-binary as a dependency, but also libraries.

@dvarrazzo
Copy link
Member

@Cykooz I appreciate your concern. And it highlights the fact that if some library started depending on that package it may cause interdependencies problem: it's something to highlight very clearly in the docs. We are a bit into uncharted waters here: we are trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it.

@dvarrazzo dvarrazzo added the wheel Bugs related to the wheel distribution, not happening installing from source label Mar 16, 2018
@goodtune
Copy link

This is a problematic solution for the reasons already mentioned - psycopg2-binary does not satisfy a psycopg2 dependency, and there are more than a few of these.

Pip already had a way to avoid using the wheel distribution - that should be the mechanism to go for a source installation, not a new package which clobbers the namespace of another.

@mmerickel
Copy link

Until PyPA adds the new "Provides" metadata there is just no good solution here that involves splitting the packages. PEP 426 actually touched on this but that PEP was since withdrawn and work on it has stalled. This means we just do not have a way to have two distributions claim the same virtual distribution name.

The options in the context of the current packaging tooling are simply

  1. stop providing wheels at all
  2. provide wheels and require users to use --no-binary if they have problems.

There are several packages including the sqlalchemy[postgresql] that expect psycopg2 and are incompatible with psycopg2-binary. Right now it's simply unresolvable if any upstream dependency of your project chooses either one... they cannot both be installed. The psycopg2 maintainers seem to think that users have control over which one they install if two upstream packages differ here, but it's actually not the case with the way pip resolves things right now. The only viable solution which puts control in the users hands is to make everything just depend on psycopg2 and have users specify --no-binary if they are having problems with the wheel.

@poswald
Copy link

poswald commented Apr 25, 2018

Can the psycopg2[nobinary] format be used to trigger a build with --no-binary or perhaps the opposite with psycopg2[binary] pulling in the bins instead of building? No idea if that's possible but it would satisfy the same dependency for the 2 versions.

@ztane
Copy link

ztane commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2. Having seen the SSL error now on Ubuntu 18.04 it seems that it is going to be here to stay.

@rouge8
Copy link

rouge8 commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2

You can specify which packages you don't want to install from wheels, e.g. --no-binary=psycopg2.

For example:

$ PIP_NO_CACHE_DIR=off pip install --no-binary=click click flask
Collecting click
  Downloading https://files.pythonhosted.org/packages/95/d9/c3336b6b5711c3ab9d1d3a80f1a3e2afeb9d8c02a7166462f6cc96570897/click-6.7.tar.gz (279kB)
    100% |████████████████████████████████| 286kB 6.6MB/s
Collecting flask
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 13.5MB/s
Collecting itsdangerous>=0.24 (from flask)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
    100% |████████████████████████████████| 51kB 12.2MB/s
Collecting Jinja2>=2.10 (from flask)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 43.8MB/s
Collecting Werkzeug>=0.14 (from flask)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
    100% |████████████████████████████████| 327kB 11.3MB/s
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask
  Running setup.py install for click ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 itsdangerous-0.24

@Vadorequest
Copy link

Vadorequest commented Jul 2, 2018

I don't understand anything about all that stuff neither really want to dive into it, I don't know what wheels are, I just know I had a working project using psycopg2, which now displays warning about being renamed, so I thought I had to migrate to psycopg2-binary, but if I remove psycopg2 then it fails connecting to my postgre DB. Kinda lost as people mentionned earlier...

Should I keep both? Should I not care and keep using psycopg2 and ignore that warning?

How do I automate pip install --no-binary :all: psycopg2 in my requirements.txt? Any way of just saying to install psycopg2 without binary and not think about it anymore?


Okay, found a solution using the following in my requirements.txt:
psycopg2==2.7.5 --no-binary psycopg2

tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
@sarusso
Copy link

sarusso commented Aug 16, 2018

@dvarrazzo you mentioned that you are "trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it", but you never commented on the alternative "--no-binary" approach. What are your thoughts?

frenzymadness added a commit to frenzymadness/django-ex that referenced this issue Aug 4, 2020
frenzymadness added a commit to frenzymadness/django-ex that referenced this issue Aug 4, 2020
@frenzymadness
Copy link

A lot of projects are affected by this change and it wasn't clear to me what happened. Thanks to this issue I know what to do.
Our case is that we have a lot of testing projects in containers hosted on OpenShift platform where we don't want to install all dependencies required to build psycopg2 from source so we have to switch all of them to psycopg2-binary.

@boolow5
Copy link

boolow5 commented Oct 12, 2020

On Ubuntu 20.04 this answer worked for me.
Answer summery:

sudo apt-get install gcc libpq-dev -y
sudo apt-get install python-dev  python-pip -y
sudo apt-get install python3-dev python3-pip python3-venv python3-wheel -y
pip3 install wheel

None of the above suggestions fixed my issue.

DavidCain added a commit to DavidCain/hgvs that referenced this issue Sep 24, 2021
This closes biocommons#623

Per the authors of the `psycopg2` package, `psycopg2-binary` "is meant
for beginners to start playing with Python and PostgreSQL without the
need to meet the build requirements."

However, it's preferable to depend on `psycopg2` for production software.

Issues in flight which should make this change irrelevant
---------------------------------------------------------

- [make data provider dependencies optional][hgvs-optional-deps]
- [replace psycopg2 with asyncpg][hgvs-asyncpg]
- [use a REST interface, eliminating libpq dependency][hgvs-rest]

------------------------------------------------------------------------

Why move away from `psycopg2-binary`?
=====================================
The psycopg2 project makes two recommendations in the
[psycopg2 vs psycopg2-binary][psycopg2-vs-psycopg2-binary] docs:

> If you are the maintainer of a published package depending on `psycopg2` you
> shouldn’t use `psycopg2-binary` as a module dependency.

Secondarily:

> **For production use you are advised to use the source distribution.**

This means that any production software depending on `hgvs` will have to
install `psycopg2-binary`.

History of the two packages
===========================
Starting with Psycopg2 2.8, there are now two packages (explained in the
[2.7.4 release notes][psycopg2-274]):

- `psycopg2`: source distribution, advised for production.
- `psycopg2-binary`: "quickest way to install" (a "pre-compiled binary version")

There is substantial discussion in [an issue on the `psycopg2` project][gh-thread],
which I won't re-summarize here, but one takeaway is that many Python packages
now depend on `psycopg2-binary`, and others on `psycopg2`.

Ideally, all libraries just depend on `psycopg2`.

Alternate solutions to the split exist (like [`Provides-Dist` from PEP 345][provides-dist]).
However, the [`Provides-Dist` field is "rarely used"][provides-dist-rarely-used]
and thus ignored most of the time.

[`hgvs` switched to `psycopg2-binary`][hgvs-switch-to-binary] shortly after the 2.7.4 release.

[gh-thread]: psycopg/psycopg2#674
[hgvs-asyncpg]: biocommons#603
[hgvs-optional-deps]: biocommons#199
[hgvs-rest]: biocommons#199 (comment)
[hgvs-switch-to-binary]: biocommons@4c75a4e5
[provides-dist-rarely-used]: https://packaging.python.org/specifications/core-metadata/#provides-dist-multiple-use
[provides-dist]: https://www.python.org/dev/peps/pep-0345/#provides-dist-multiple-use
[psycopg2-274]: https://www.psycopg.org/articles/2018/02/08/psycopg-274-released/
[psycopg2-vs-psycopg2-binary]: https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary
reece pushed a commit to biocommons/hgvs that referenced this issue Sep 25, 2021
This closes #623

Per the authors of the `psycopg2` package, `psycopg2-binary` "is meant
for beginners to start playing with Python and PostgreSQL without the
need to meet the build requirements."

However, it's preferable to depend on `psycopg2` for production software.

Issues in flight which should make this change irrelevant
---------------------------------------------------------

- [make data provider dependencies optional][hgvs-optional-deps]
- [replace psycopg2 with asyncpg][hgvs-asyncpg]
- [use a REST interface, eliminating libpq dependency][hgvs-rest]

------------------------------------------------------------------------

Why move away from `psycopg2-binary`?
=====================================
The psycopg2 project makes two recommendations in the
[psycopg2 vs psycopg2-binary][psycopg2-vs-psycopg2-binary] docs:

> If you are the maintainer of a published package depending on `psycopg2` you
> shouldn’t use `psycopg2-binary` as a module dependency.

Secondarily:

> **For production use you are advised to use the source distribution.**

This means that any production software depending on `hgvs` will have to
install `psycopg2-binary`.

History of the two packages
===========================
Starting with Psycopg2 2.8, there are now two packages (explained in the
[2.7.4 release notes][psycopg2-274]):

- `psycopg2`: source distribution, advised for production.
- `psycopg2-binary`: "quickest way to install" (a "pre-compiled binary version")

There is substantial discussion in [an issue on the `psycopg2` project][gh-thread],
which I won't re-summarize here, but one takeaway is that many Python packages
now depend on `psycopg2-binary`, and others on `psycopg2`.

Ideally, all libraries just depend on `psycopg2`.

Alternate solutions to the split exist (like [`Provides-Dist` from PEP 345][provides-dist]).
However, the [`Provides-Dist` field is "rarely used"][provides-dist-rarely-used]
and thus ignored most of the time.

[`hgvs` switched to `psycopg2-binary`][hgvs-switch-to-binary] shortly after the 2.7.4 release.

[gh-thread]: psycopg/psycopg2#674
[hgvs-asyncpg]: #603
[hgvs-optional-deps]: #199
[hgvs-rest]: #199 (comment)
[hgvs-switch-to-binary]: 4c75a4e5
[provides-dist-rarely-used]: https://packaging.python.org/specifications/core-metadata/#provides-dist-multiple-use
[provides-dist]: https://www.python.org/dev/peps/pep-0345/#provides-dist-multiple-use
[psycopg2-274]: https://www.psycopg.org/articles/2018/02/08/psycopg-274-released/
[psycopg2-vs-psycopg2-binary]: https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary
mchesterkadwell added a commit to Multimedia-Avesta/django_main_muya_wce that referenced this issue Feb 3, 2022
VV-YY added a commit to VV-YY/django-ex that referenced this issue Jan 17, 2023
sperez927 pushed a commit to sperez927/django-moretraining that referenced this issue Nov 11, 2023
sperez927 added a commit to sperez927/django-moretraining that referenced this issue Nov 30, 2023
tanaypf9 pushed a commit to tanaypf9/pf9-requirements that referenced this issue May 20, 2024
Patch Set 1:

> Right, but that means coinstallability is going to be difficult.
 > Unless we switch globally this means all non-containerized
 > deployments have to edit requirements to.match one of them
 > globally.

Hmm, so there's a more detailed discussion on GitHub [1] and from the looks of things, this package split was intended to work issues with SSL libraries. Unfortunately they don't discuss a resolution to potential clobbering issues, likely because there isn't one. I've started a discussion on the mailing list [2] but at this time it seems we might be better off just requiring the installation of libpq once again. I'm going to abandon this until we have a better idea of how to approach this.

[1] psycopg/psycopg2#674
[2] http://lists.openstack.org/pipermail/openstack-dev/2018-May/130497.html

Patch-set: 1
Reviewer: Gerrit User 15334 <15334@4a232e18-c5a9-48ee-94c0-e04e7cca6543>
Label: Verified=0
crveskig added a commit to crveskig/Django-Project that referenced this issue May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wheel Bugs related to the wheel distribution, not happening installing from source
Projects
None yet
Development

No branches or pull requests