Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

psycopg2-binary: Why? #674

Closed
jleclanche opened this issue Feb 11, 2018 · 33 comments
Closed

psycopg2-binary: Why? #674

jleclanche opened this issue Feb 11, 2018 · 33 comments
Labels

Comments

@jleclanche
Copy link

@jleclanche jleclanche commented Feb 11, 2018

Forgive the blunt question but I cannot find discussion on why the split of psycopg2 and psycopg2-binary is happening. The current (2.7) way of doing wheel+source distribution works exactly how it's supposed to. Splitting into a separate -binary distribution actually breaks any library that depends on psycopg2=>2.x, which includes for example sqlalchemy's postgres bindings, because setuptools will complain the dependency is not matched.

@dvarrazzo
Copy link
Member

@dvarrazzo dvarrazzo commented Feb 11, 2018

The discussion is here

The explanation is here

If you have another solution let us know in the mailing list. The solution should involve not segfaulting.

@dvarrazzo dvarrazzo closed this Feb 11, 2018
@jleclanche
Copy link
Author

@jleclanche jleclanche commented Feb 13, 2018

Thanks for the context! This is super helpful.
I actually hit the issue specified there (uwsgi+psycopg2 libssl issues on debian). It's tough because I get that this stuff should work out of the box, but at the same time I think the current solution will only work out of the box as far as people won't be using psycopg2-binary.

If you are affected by that particular issue, compiling from source is one of the more appropriate fixes and pip has a command line flag for it (which psycopg2 itself documented in various places including this issue tracker).

I understand the desire to fix the issue (as I said, I'm affected by it), but I don't think this really fixes anything:

  • Users who stay on psycopg2 now have a runtimewarning they may or may not understand
  • pip install psycopg2 time and requirements changed with no clear explanation why without knowing the background or having seen aforementioned warning
  • Users who switch to psycopg2-binary now have issues with various package requires such as the one I mentioned in this issue (sqlalchemy is a big user of psycopg2, this is probably going to surface a lot more once 2.8 is released)
  • And for the users who switch to psycopg2-binary, the original issue with libssl still exists. It's just been moved.

I think the "solution" really is the --no-binary pip flag. Unless I completely misunderstand something, that has the same effect as moving the wheels to a different package, except that for this package the wheels will not have added issues.

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

I fully agree with @jleclanche. Creating of psycopg2-binary is not a solution for end users. On the contrary, it is a new problem for users whom uses many third-party libraries depended from psycopg2. Now one half of this libraries depends from psycopg2 (e.g. sqlalchemy), and other half - from psycopg2-binary (e.g. pgcli). Which version of psycopg2 package will be installed and used in that situation?

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

psycopg2-binary has not resolved problem with libssl.

Imagine, what if near future all python libraries will be depends from psycopg2-binary instead of psycopg2. What will this situation be different from when there was only one version of psycopg2?

@dvarrazzo
Copy link
Member

@dvarrazzo dvarrazzo commented Mar 7, 2018

@Cykooz use psycopg2 and forget the wheel for your use. Let the other decide for theirs. If they are writing multithread programs using ssh they better use psycopg2. Otherwise they are fine with -wheels. The libssl one is not a solvable problem. If it is a problem for you forget the wheels.

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

But not all depends from me. I use different third-party packages which already has psycopg2 or psycopg2-binary in self dependencies. And I can't control an order of installation of this packages. Sometimes psycopg2 will be installed first, other time - psycopg2-binary will.

By default pip don't use package name from METADATA to check that it already installed. It compare only package names taken from names of wheel files. This means that I can install both packages at the same time.

If I install psycopg2-binary first, and then install psycopg2:

$ pip install psycopg2-binary
$ pip install psycopg2

then if I uninstall psycopg2-binary:

$ pip uninstall psycopg2-binary

this will completely remove the code of psycopg2 from my python:

$ python
>>> import psycopg2
...
ImportError: No module named psycopg2

But pip will still consider that the psycopg2 is installed.

$ pip install psycopg2
Requirement already satisfied: psycopg2 in ./dist-packages

All of this is not very well.

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

Third-party packages should not determine which version of the psycopg2 distributive should be used. Because they do not know in what environment they will be used. Version of distributive of psycopg2 should be determined solely by external tools. For example, with help of the --no-binary option for pip.

@dvarrazzo
Copy link
Member

@dvarrazzo dvarrazzo commented Mar 7, 2018

@Cykooz What third party package depend on psycopg2-binary?

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

@dvarrazzo pgcli depends from psycopg2-binary.

@dvarrazzo
Copy link
Member

@dvarrazzo dvarrazzo commented Mar 7, 2018

@Cykooz pgcli is an interactive tool: it doesn't strike me as something that has to live 100% in the virtualenv of e.g. a Django webapp using psycopg. But those kind of tools are exactly the reason for which I don't want to get rid of the wheels altogether. It's probably worth a chat with them to understand what's the best strategy.

@Cykooz
Copy link

@Cykooz Cykooz commented Mar 7, 2018

Oh, maybe you are in something right.

I'm just afraid that not only standalone applications will include psycopg2-binary as a dependency, but also libraries.

@dvarrazzo
Copy link
Member

@dvarrazzo dvarrazzo commented Mar 7, 2018

@Cykooz I appreciate your concern. And it highlights the fact that if some library started depending on that package it may cause interdependencies problem: it's something to highlight very clearly in the docs. We are a bit into uncharted waters here: we are trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it.

@goodtune
Copy link

@goodtune goodtune commented Mar 23, 2018

This is a problematic solution for the reasons already mentioned - psycopg2-binary does not satisfy a psycopg2 dependency, and there are more than a few of these.

Pip already had a way to avoid using the wheel distribution - that should be the mechanism to go for a source installation, not a new package which clobbers the namespace of another.

@mmerickel
Copy link

@mmerickel mmerickel commented Apr 13, 2018

Until PyPA adds the new "Provides" metadata there is just no good solution here that involves splitting the packages. PEP 426 actually touched on this but that PEP was since withdrawn and work on it has stalled. This means we just do not have a way to have two distributions claim the same virtual distribution name.

The options in the context of the current packaging tooling are simply

  1. stop providing wheels at all
  2. provide wheels and require users to use --no-binary if they have problems.

There are several packages including the sqlalchemy[postgresql] that expect psycopg2 and are incompatible with psycopg2-binary. Right now it's simply unresolvable if any upstream dependency of your project chooses either one... they cannot both be installed. The psycopg2 maintainers seem to think that users have control over which one they install if two upstream packages differ here, but it's actually not the case with the way pip resolves things right now. The only viable solution which puts control in the users hands is to make everything just depend on psycopg2 and have users specify --no-binary if they are having problems with the wheel.

@poswald
Copy link

@poswald poswald commented Apr 25, 2018

Can the psycopg2[nobinary] format be used to trigger a build with --no-binary or perhaps the opposite with psycopg2[binary] pulling in the bins instead of building? No idea if that's possible but it would satisfy the same dependency for the 2 versions.

@ztane
Copy link

@ztane ztane commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2. Having seen the SSL error now on Ubuntu 18.04 it seems that it is going to be here to stay.

@rouge8
Copy link

@rouge8 rouge8 commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2

You can specify which packages you don't want to install from wheels, e.g. --no-binary=psycopg2.

For example:

$ PIP_NO_CACHE_DIR=off pip install --no-binary=click click flask
Collecting click
  Downloading https://files.pythonhosted.org/packages/95/d9/c3336b6b5711c3ab9d1d3a80f1a3e2afeb9d8c02a7166462f6cc96570897/click-6.7.tar.gz (279kB)
    100% |████████████████████████████████| 286kB 6.6MB/s
Collecting flask
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 13.5MB/s
Collecting itsdangerous>=0.24 (from flask)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
    100% |████████████████████████████████| 51kB 12.2MB/s
Collecting Jinja2>=2.10 (from flask)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 43.8MB/s
Collecting Werkzeug>=0.14 (from flask)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
    100% |████████████████████████████████| 327kB 11.3MB/s
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask
  Running setup.py install for click ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 itsdangerous-0.24
@Vadorequest
Copy link

@Vadorequest Vadorequest commented Jul 2, 2018

I don't understand anything about all that stuff neither really want to dive into it, I don't know what wheels are, I just know I had a working project using psycopg2, which now displays warning about being renamed, so I thought I had to migrate to psycopg2-binary, but if I remove psycopg2 then it fails connecting to my postgre DB. Kinda lost as people mentionned earlier...

Should I keep both? Should I not care and keep using psycopg2 and ignore that warning?

How do I automate pip install --no-binary :all: psycopg2 in my requirements.txt? Any way of just saying to install psycopg2 without binary and not think about it anymore?


Okay, found a solution using the following in my requirements.txt:
psycopg2==2.7.5 --no-binary psycopg2

tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018
@sarusso
Copy link

@sarusso sarusso commented Aug 16, 2018

@dvarrazzo you mentioned that you are "trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it", but you never commented on the alternative "--no-binary" approach. What are your thoughts?

frenzymadness added a commit to frenzymadness/django-ex that referenced this issue Aug 4, 2020
frenzymadness added a commit to frenzymadness/django-ex that referenced this issue Aug 4, 2020
@frenzymadness
Copy link

@frenzymadness frenzymadness commented Aug 4, 2020

A lot of projects are affected by this change and it wasn't clear to me what happened. Thanks to this issue I know what to do.
Our case is that we have a lot of testing projects in containers hosted on OpenShift platform where we don't want to install all dependencies required to build psycopg2 from source so we have to switch all of them to psycopg2-binary.

@boolow5
Copy link

@boolow5 boolow5 commented Oct 12, 2020

On Ubuntu 20.04 this answer worked for me.
Answer summery:

sudo apt-get install gcc libpq-dev -y
sudo apt-get install python-dev  python-pip -y
sudo apt-get install python3-dev python3-pip python3-venv python3-wheel -y
pip3 install wheel

None of the above suggestions fixed my issue.

jonathangreen added a commit to lyrasis/simplye-server-core that referenced this issue Mar 8, 2021
One for development, one for production and one containing the common
components for each of them.

Having seperate requirements for dev and production lets us only install
the packages used for testing into the test environment.

The reason we have a common "base" requirements file is that the
psycopg2 package has two versions (psycopg2-binary and psycopg2)[1][2].
The -binary version isn't recommended for production use, but is very
helpful for testing, so we don't have to build the psycopg2 package
every time CI runs.

[1] - psycopg/psycopg2#674
[2] - https://www.psycopg.org/articles/2018/02/08/psycopg-274-released/
jonathangreen added a commit to lyrasis/simplye-server-core that referenced this issue Mar 8, 2021
One for development, one for production and one containing the common
components for each of them.

Having seperate requirements for dev and production lets us only install
the packages used for testing into the test environment.

The reason we have a common "base" requirements file is that the
psycopg2 package has two versions (psycopg2-binary and psycopg2)[1][2].
The -binary version isn't recommended for production use, but is very
helpful for testing, so we don't have to build the psycopg2 package
every time CI runs.

[1] - psycopg/psycopg2#674
[2] - https://www.psycopg.org/articles/2018/02/08/psycopg-274-released/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet