psycopg2-binary warning

[gnilrets]

I just upgraded pip to v 10.0.1 (from 9.x) and various packages and now I'm getting the following warning:

/opt/conda/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For detai
ls see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.                   

psycopg2 seems to be a dependency for sqlalchemy-redshift, so even if I only put psycopg2-binary in my requirements.in file, sqlalchemy-redshift will build using psycopg2 and I still get the warning above.

[jklukas]

It's not clear to me that there's any elegant way of handling this generally. I think we may need to simply wait out the next months while the default behavior for latest psycopg2 emits a warning.

For the time being, that puts some extra burden on developers to figure out what they want to do. In your case, if you want to use the binary package, you'd need to tell pip to explicitly exclude psycopg2.

If you have suggestions for a better general solution, I'd be very interested to hear.

[calebcauthon]

@jklukas Pardon my ignorance but why can't we just update setup.py to use psycopg2-binary? And what happens after we wait out the next few months?

[jklukas]

Pardon my ignorance

I don't fully understand the implications of this psycopg2 packaging change either, so let's learn together.

can't we just update setup.py to use psycopg2-binary?

Whether we update to psycopg2-binary or keep things as is, we'll be making a choice for the user about whether to default to installing from source or from the pre-built wheel. There's some discussion in http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/ about the broader issue.

My understanding is that with the release of psycopg2 2.8.0, the warning will go away and the psycopg2 dependency will always install from source, which seems the more conservative choice. Users will need to exclude psycopg2 and install psycopg2-binary instead if they want to use the pre-built wheel.

[oakhan3]

@jklukas since psycopg2 has decided to fragment their releases and allows users to specify which "edition" they want, it would make sense to me if the psycopg2 requirement was removed altogether from SQLAlchemy-Redshift. If it can be helped, the package shouldn't bias one edition over the other.

pip doesnt really allow us to say require one or the other though, maybe a new runtime check could be included -

>>> import sqlalchemy_redshift
Traceback (most recent call last):
  File "<...>", line 1, in <module>
ImportError: psycopg2 or psycopg2-binary must be explicitly installed prior to using SQLAlchemy-Redshift

and EXTRAS requirements for each could be added if SQLAlchemy-Redshift depends on some minimal lower bound of psycopg2 (it looks to be psycopg2>2.7.3.2 atm)

^^ All of this with a bump up to the minor version as it is somewhat breaking of course.

[jklukas]

I think that approach is reasonable, @oakhan3, but it's not clear to me whether or not it's a better overall user experience. It trades one surprising behavior (warning during install) for a different surprising behavior (error on import due to a missing dependency) that also involves making the setup code more complex.

I'd want to see more justification for the change. Do we have examples of how other projects that depend on psycopg2 are handling this problem?

[oakhan3]

Good point @jklukas, unfortunately, it seems to be a deeply debated topic with no clean end result - psycopg/psycopg2#674

2 big projects that kind of addressed this -

pgcli

• pgcli prefers to install "psycopg2" and instructs others to install from source if they see issues arise - https://github.com/dbcli/pgcli#quick-start
• They also silenced the warning, see this commit - dbcli/pgcli@b8a6dfd

SQLAlchemy

• SQLAlchemy went the separate extras route (if you choose to specify the adapter install via SQLAlchemy install) - https://github.com/sqlalchemy/sqlalchemy/blob/master/setup.py#L167

It seems like keeping it as-is is the best option here - I will probably silence the warning or install from source in my own projects similar to how pgcli does it under the hood - it doesn't seem like a decision any library should make for me

[jklukas]

I do hope that some of the pain goes away once 2.8.0 is released and no longer emits a warning about the deprecation for cases where users are letting pip install latest.

[jklukas]

I think we're past the problematic stage here and users are unlikely to see this warning with current versions of psycopg2