New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

psycopg2-binary: Why? #674

Closed
jleclanche opened this Issue Feb 11, 2018 · 27 comments

Comments

Projects
None yet
@jleclanche

jleclanche commented Feb 11, 2018

Forgive the blunt question but I cannot find discussion on why the split of psycopg2 and psycopg2-binary is happening. The current (2.7) way of doing wheel+source distribution works exactly how it's supposed to. Splitting into a separate -binary distribution actually breaks any library that depends on psycopg2=>2.x, which includes for example sqlalchemy's postgres bindings, because setuptools will complain the dependency is not matched.

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Feb 11, 2018

The discussion is here

The explanation is here

If you have another solution let us know in the mailing list. The solution should involve not segfaulting.

@dvarrazzo dvarrazzo closed this Feb 11, 2018

@jleclanche

This comment has been minimized.

jleclanche commented Feb 13, 2018

Thanks for the context! This is super helpful.
I actually hit the issue specified there (uwsgi+psycopg2 libssl issues on debian). It's tough because I get that this stuff should work out of the box, but at the same time I think the current solution will only work out of the box as far as people won't be using psycopg2-binary.

If you are affected by that particular issue, compiling from source is one of the more appropriate fixes and pip has a command line flag for it (which psycopg2 itself documented in various places including this issue tracker).

I understand the desire to fix the issue (as I said, I'm affected by it), but I don't think this really fixes anything:

  • Users who stay on psycopg2 now have a runtimewarning they may or may not understand
  • pip install psycopg2 time and requirements changed with no clear explanation why without knowing the background or having seen aforementioned warning
  • Users who switch to psycopg2-binary now have issues with various package requires such as the one I mentioned in this issue (sqlalchemy is a big user of psycopg2, this is probably going to surface a lot more once 2.8 is released)
  • And for the users who switch to psycopg2-binary, the original issue with libssl still exists. It's just been moved.

I think the "solution" really is the --no-binary pip flag. Unless I completely misunderstand something, that has the same effect as moving the wheels to a different package, except that for this package the wheels will not have added issues.

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

I fully agree with @jleclanche. Creating of psycopg2-binary is not a solution for end users. On the contrary, it is a new problem for users whom uses many third-party libraries depended from psycopg2. Now one half of this libraries depends from psycopg2 (e.g. sqlalchemy), and other half - from psycopg2-binary (e.g. pgcli). Which version of psycopg2 package will be installed and used in that situation?

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

psycopg2-binary has not resolved problem with libssl.

Imagine, what if near future all python libraries will be depends from psycopg2-binary instead of psycopg2. What will this situation be different from when there was only one version of psycopg2?

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Mar 7, 2018

@Cykooz use psycopg2 and forget the wheel for your use. Let the other decide for theirs. If they are writing multithread programs using ssh they better use psycopg2. Otherwise they are fine with -wheels. The libssl one is not a solvable problem. If it is a problem for you forget the wheels.

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

But not all depends from me. I use different third-party packages which already has psycopg2 or psycopg2-binary in self dependencies. And I can't control an order of installation of this packages. Sometimes psycopg2 will be installed first, other time - psycopg2-binary will.

By default pip don't use package name from METADATA to check that it already installed. It compare only package names taken from names of wheel files. This means that I can install both packages at the same time.

If I install psycopg2-binary first, and then install psycopg2:

$ pip install psycopg2-binary
$ pip install psycopg2

then if I uninstall psycopg2-binary:

$ pip uninstall psycopg2-binary

this will completely remove the code of psycopg2 from my python:

$ python
>>> import psycopg2
...
ImportError: No module named psycopg2

But pip will still consider that the psycopg2 is installed.

$ pip install psycopg2
Requirement already satisfied: psycopg2 in ./dist-packages

All of this is not very well.

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

Third-party packages should not determine which version of the psycopg2 distributive should be used. Because they do not know in what environment they will be used. Version of distributive of psycopg2 should be determined solely by external tools. For example, with help of the --no-binary option for pip.

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Mar 7, 2018

@Cykooz What third party package depend on psycopg2-binary?

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

@dvarrazzo pgcli depends from psycopg2-binary.

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Mar 7, 2018

@Cykooz pgcli is an interactive tool: it doesn't strike me as something that has to live 100% in the virtualenv of e.g. a Django webapp using psycopg. But those kind of tools are exactly the reason for which I don't want to get rid of the wheels altogether. It's probably worth a chat with them to understand what's the best strategy.

@Cykooz

This comment has been minimized.

Cykooz commented Mar 7, 2018

Oh, maybe you are in something right.

I'm just afraid that not only standalone applications will include psycopg2-binary as a dependency, but also libraries.

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Mar 7, 2018

@Cykooz I appreciate your concern. And it highlights the fact that if some library started depending on that package it may cause interdependencies problem: it's something to highlight very clearly in the docs. We are a bit into uncharted waters here: we are trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it.

@omad omad referenced this issue Mar 13, 2018

Merged

Initial docker setup in core ODC repo #382

2 of 2 tasks complete

@dvarrazzo dvarrazzo added the wheel label Mar 16, 2018

@goodtune

This comment has been minimized.

goodtune commented Mar 23, 2018

This is a problematic solution for the reasons already mentioned - psycopg2-binary does not satisfy a psycopg2 dependency, and there are more than a few of these.

Pip already had a way to avoid using the wheel distribution - that should be the mechanism to go for a source installation, not a new package which clobbers the namespace of another.

@mmerickel

This comment has been minimized.

mmerickel commented Apr 13, 2018

Until PyPA adds the new "Provides" metadata there is just no good solution here that involves splitting the packages. PEP 426 actually touched on this but that PEP was since withdrawn and work on it has stalled. This means we just do not have a way to have two distributions claim the same virtual distribution name.

The options in the context of the current packaging tooling are simply

  1. stop providing wheels at all
  2. provide wheels and require users to use --no-binary if they have problems.

There are several packages including the sqlalchemy[postgresql] that expect psycopg2 and are incompatible with psycopg2-binary. Right now it's simply unresolvable if any upstream dependency of your project chooses either one... they cannot both be installed. The psycopg2 maintainers seem to think that users have control over which one they install if two upstream packages differ here, but it's actually not the case with the way pip resolves things right now. The only viable solution which puts control in the users hands is to make everything just depend on psycopg2 and have users specify --no-binary if they are having problems with the wheel.

@poswald

This comment has been minimized.

poswald commented Apr 25, 2018

Can the psycopg2[nobinary] format be used to trigger a build with --no-binary or perhaps the opposite with psycopg2[binary] pulling in the bins instead of building? No idea if that's possible but it would satisfy the same dependency for the 2 versions.

@ztane

This comment has been minimized.

ztane commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2. Having seen the SSL error now on Ubuntu 18.04 it seems that it is going to be here to stay.

@rouge8

This comment has been minimized.

rouge8 commented Jun 26, 2018

However the --no-binary is hardly a solution either, because with long requirements lists many other packages are fine to be installed from wheels except psycopg2

You can specify which packages you don't want to install from wheels, e.g. --no-binary=psycopg2.

For example:

$ PIP_NO_CACHE_DIR=off pip install --no-binary=click click flask
Collecting click
  Downloading https://files.pythonhosted.org/packages/95/d9/c3336b6b5711c3ab9d1d3a80f1a3e2afeb9d8c02a7166462f6cc96570897/click-6.7.tar.gz (279kB)
    100% |████████████████████████████████| 286kB 6.6MB/s
Collecting flask
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 13.5MB/s
Collecting itsdangerous>=0.24 (from flask)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
    100% |████████████████████████████████| 51kB 12.2MB/s
Collecting Jinja2>=2.10 (from flask)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 43.8MB/s
Collecting Werkzeug>=0.14 (from flask)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
    100% |████████████████████████████████| 327kB 11.3MB/s
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask
  Running setup.py install for click ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 itsdangerous-0.24
@Vadorequest

This comment has been minimized.

Vadorequest commented Jul 2, 2018

I don't understand anything about all that stuff neither really want to dive into it, I don't know what wheels are, I just know I had a working project using psycopg2, which now displays warning about being renamed, so I thought I had to migrate to psycopg2-binary, but if I remove psycopg2 then it fails connecting to my postgre DB. Kinda lost as people mentionned earlier...

Should I keep both? Should I not care and keep using psycopg2 and ignore that warning?

How do I automate pip install --no-binary :all: psycopg2 in my requirements.txt? Any way of just saying to install psycopg2 without binary and not think about it anymore?


Okay, found a solution using the following in my requirements.txt:
psycopg2==2.7.5 --no-binary psycopg2

tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018

tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018

tyacbovi added a commit to cloudify-cosmo/cloudify-manager that referenced this issue Aug 13, 2018

@sarusso

This comment has been minimized.

sarusso commented Aug 16, 2018

@dvarrazzo you mentioned that you are "trying to figure out if the idea behind the psycopg2-binary is any good or if it's just the case of giving up with it", but you never commented on the alternative "--no-binary" approach. What are your thoughts?

@randygrolemund

This comment was marked as disruptive content.

randygrolemund commented Aug 19, 2018

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Aug 20, 2018

I apologise for the noise: I was away for a few days and didn't see this moronic thread. I will delete the crap left by @randygrolemund which I have blocked so I trust he will use stackoverflow to show up his arrogance and ignorance from now on.

@psycopg psycopg deleted a comment from randygrolemund Aug 20, 2018

@psycopg psycopg deleted a comment from mmerickel Aug 20, 2018

@psycopg psycopg deleted a comment from randygrolemund Aug 20, 2018

@psycopg psycopg deleted a comment from randygrolemund Aug 20, 2018

@psycopg psycopg deleted a comment from randygrolemund Aug 20, 2018

@psycopg psycopg deleted a comment from goodtune Aug 20, 2018

@psycopg psycopg deleted a comment from randygrolemund Aug 20, 2018

@psycopg psycopg deleted a comment from sarusso Aug 20, 2018

@psycopg psycopg deleted a comment from jleclanche Aug 20, 2018

@psycopg psycopg deleted a comment from Vadorequest Aug 20, 2018

oyvindkolbu added a commit to usit-gd/mreg that referenced this issue Oct 3, 2018

Update to latest Django and deps.
While here, add psycopg2-binary, per
psycopg/psycopg2#674
@mlissner

This comment has been minimized.

mlissner commented Oct 29, 2018

Just throwing this out there as user experience feedback. I just spent 30 minutes trying to figure out what I should do about the warning that popped upon my console when we upgraded to psycopg 2.7.

I didn't know whether I had a binary thing or a source thing installed, or why the split, or what. All I knew is that I have a warning that I need to somehow fix. Seems that if I'm happy building things myself, I can do:

psycopg2>=2.7,<2.8 --no-binary psycopg2

So I did that. Alas, I still get the error on the console. So, I went to check if Django supports version 2.8, and it just says anything > 2.5 is fine. Great! So I update my requirements.txt file to have psychopg~=2.8, and...version 2.8 isn't yet out.

So my options are:

  1. Live with the warning (I choose this)

  2. Install the psycopg-binary package, except that's for toy projects, I guess. ("The -binary package is meant for beginners to start playing with Python and PostgreSQL without the need to meet the build requirements.")

I'm outside my comfort zone here, but this all seems pretty funky.

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Oct 30, 2018

@mlissner sorry for the bad experience. psycopg2>=2.7,<2.8 --no-binary psycopg2 should work. You may have the binary package still there. Just delete it, with pip uninstall, and if it doesn't come away, find the package (import psycopg2; print psycopg2) and delete the directory.

@mlissner

This comment has been minimized.

mlissner commented Oct 30, 2018

OK, uninstalling and then reinstalling with the --no-binary flag did it. Thanks for the tip. So if I'm understanding correctly, anybody that has a serious project that wants to upgrade psycopg and not have the warning emitted will need to take this step. Should this go in the docs?

Also worth mentioning, now we're in the realm where upgrading psycopg properly seems to mean writing an upgrade script to uninstall the old one and reinstall the new one (since requirements.txt doesn't do uninstallations). There's probably a way around it, but this also seems to mean taking DB clients offline temporarily too during the transition.

@merwok

This comment has been minimized.

merwok commented Oct 30, 2018

In my team, having --no-binary in the requirements complicated local setup for some developers (i.e. people using Mac OS and its ten ways of installing stuff).

(We really want to build from source against the system libraries for deployments, rather than having pre-built duplicated libs in a wheel.)

Devs have to ignore the requirements files even if they first install psycopg2-binary, because the latter does not Provides-Dist: psycopg2. I’m not 100% sure that pip respects that bit of metadata but this could be worth checking?

@dvarrazzo

This comment has been minimized.

Member

dvarrazzo commented Oct 30, 2018

Happy to fix provides-dist if it's any useful. Is there documentation for this feature?

@merwok

This comment has been minimized.

merwok commented Oct 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment