Wheel support for linux aarch64

[odidev]

Summary
Installing psycopg2-binary on aarch64 via pip using command "pip3 install psycopg2-binary" tries to build wheel from source code

Problem description
psycopg2-binary don't have wheel for aarch64 on PyPI repository. So, while installing psycopg2-binary via pip on aarch64, pip builds wheel for same resulting in it takes more time to install psycopg2-binary. Making wheel available for aarch64 will benefit aarch64 users by minimizing psycopg2-binary installation time.

Expected Output
Pip should be able to download psycopg2-binary wheel from PyPI repository rather than building it from source code.

@psycopg2-binary-team, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make psycopg2-binary wheel available for aarch64. It will be a great opportunity for me to work with you.

[dvarrazzo]

Looking at https://github.com/pypa/manylinux it seems that aarch64 is supported in manylinux14. Currently psycopg is built with manylinux1.

No problem if you would like to extend the build system to cover aarch64.

[odidev]

Ping @dvarrazzo

I have added aarch64 support for travis build. The build is successful and all the test cases are passing.
Following is the link of travis file changes:
https://github.com/odidev/psycopg2/blob/master/.travis.yml

Below is the link of the travis build:
https://travis-ci.org/github/odidev/psycopg2/builds/705776806

Please review and let me know about the next step.

Thanks

[dvarrazzo]

Hello @odidev

Please open a branch instead of working on master, so you can create a merge request.

You have deleted the tasks for the x86 build: that's ok while you work but of course they will have to be reinstated before merging.

Your tests didn't run: see https://travis-ci.org/github/odidev/psycopg2/jobs/705776816

What you have done doesn't create any package: packages are created in the https://github.com/psycopg/psycopg2-wheels repos. I expect that a merge request will be produced there.

As you are addressing new packages, you can probably support only from python 3.5 on and avoid unsupported Python versions such as 2.7 and 3.4

[odidev]

@dvarrazzo ,

I have updated the Travis and added arm64 jobs from python versions 3.5 and resolved test issues. Please review changes required for psycopg2 in odidev@1fd9942

Psycopg2 Report: https://travis-ci.org/github/odidev/psycopg2/builds/707942496

As suggested, I have added arm64 jobs in travis in psycopg2-wheels. Please review changes required for psycopg2-wheels in odidev/psycopg2-wheels@1e02989

Psycopg2-wheels Report: https://travis-ci.org/github/odidev/psycopg2-wheels/builds/709916473

Can I raise PR or let me know if any more changes required?

[dvarrazzo]

Hello @odidev, thank you for your work so far.

I see a few things that can be fixed. If you open a MR (two, one for each repos) it would be easier to comment on them.

[odidev]

@dvarrazzo,

Thanks for the reply.

Sure, I will raise PR in 2 repos.

[dvarrazzo]

Closed via #1130 and psycopg/psycopg2-wheels#14. Thank you very much!

[pg1671]

Hi, I am testing the AWS T4g instances for our platform and installing psycopg2-binary still tries to build from source.

Collecting psycopg2-binary
Using cached psycopg2-binary-2.8.6.tar.gz (384 kB)
ERROR: Command errored out with exit status 1:
command: /djangoenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-glh17wc0/psycopg2-binary/setup.py'"'"'; file='"'"'/tmp/pip-install-glh17wc0/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info
cwd: /tmp/pip-install-glh17wc0/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating /tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info
writing /tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file '/tmp/pip-install-glh17wc0/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'

Error: pg_config executable not found.

pg_config is required to build psycopg2 from source.  Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:

    python setup.py build_ext --pg-config /path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'.

If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.

For further information please check the 'doc/src/install.rst' file (also at
<https://www.psycopg.org/docs/install.html>).

----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Cheers

[dvarrazzo]

With what version of pip? And using some --verbose why does pip refuse to install the package available on PyPI?

[pg1671]

thanks daniele,

I am using python 3.8.5 and pip 20.0.2.

Upgrading to the latest pip 21.0.1 does the same thing, but strangely loops over older (and newer) versions and also fails?

When I run will verbose I get lots of:
Skipping link: none of the wheel's tags match:

An example below:

Skipping link: none of the wheel's tags match: cp38-cp38-manylinux1_i686: https://files.pythonhosted.org/packages/b8/63/06e08c2c19737c305777806113491f0b2beeec94a3cf4844844ea1c20f86/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_i686.whl#sha256=a0c50db33c32594305b0ef9abc0cb7db13de7621d2cadf8392a1d9b3c437ef77 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.)
Skipping link: none of the wheel's tags match: cp38-cp38-manylinux1_x86_64: https://files.pythonhosted.org/packages/b5/5a/985969fb49617701ddf10b8c6b3f53a2ffd7feaaf598277a94d62401c4d9/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl#sha256=2dac98e85565d5688e8ab7bdea5446674a83a3945a8f416ad0110018d1501b94 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.)
Skipping link: none of the wheel's tags match: cp38-cp38-win32: https://files.pythonhosted.org/packages/02/1b/549967f3ce5f6e3155380728dbb9d2844c286c7408e734db88d7b9c65847/psycopg2_binary-2.8.6-cp38-cp38-win32.whl#sha256=bd1be66dde2b82f80afb9459fc618216753f67109b859a361cf7def5c7968729 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.)
Skipping link: none of the wheel's tags match: cp38-cp38-win_amd64: https://files.pythonhosted.org/packages/90/20/25f1ab5810e1956029b8941c938672bbd4c72bdbe6ae14d68d59aaf24f96/psycopg2_binary-2.8.6-cp38-cp38-win_amd64.whl#sha256=8cd0fb36c7412996859cb4606a35969dd01f4ea34d9812a141cd920c3b18be77 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.)
Skipping link: none of the wheel's tags match: cp39-cp39-macosx_10_10_intel, cp39-cp39-macosx_10_10_x86_64, cp39-cp39-macosx_10_9_intel, cp39-cp39-macosx_10_9_x86_64: https://files.pythonhosted.org/packages/5e/23/58c3e226c878f75fe5887b27dd4539dd6f9861f874139eadc01f0c4ee40f/psycopg2_binary-2.8.6-cp39-cp39-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=89705f45ce07b2dfa806ee84439ec67c5d9a0ef20154e0e475e2b2ed392a5b83 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.)

It seems to be there is no wheel for arm64 (aarch64) and the wheels that are published are still built with manylinux1 (which is mentioned in the description of this issue as not supporting arm64).

Have updated wheels been published?

[pg1671]

I would expect there to a manylinux2014_aarch64 wheel.

[dvarrazzo]

Right. No: the packages for arm is not released yet: see psycopg/psycopg2-wheels#15

[pg1671]

hi, thanks for the clarification. Any chance of a dot release to update the wheels?

The arm based instances on ec2 are looking very good for value and performance, surprised more users aren't hitting this issue.

[dvarrazzo]

psycopg2-binary is not advised for production, so many would have just followed the advice to install psycopg2 from source.

I will not able to make a release until at least the end of April as I have to focus on other task. My personal suggestion is that you don't use the -binary package on AWS.

[pg1671]

I saw that you don't recommend binary for production. Can I ask why?

Will look at building from source.

[dvarrazzo]

I saw that you don't recommend binary for production. Can I ask why?

Here are a few of the problems wheel distributions have caused.

The main problem was the coexistence of the system libssl (e.g. because using urllib to open an https website) and the wheel-packaged libssl, see especially #543

These problems might have been resolved moving to libssl 1.1, however I have never been able to reproduce the bug myself and so to test if something is really a solution. As such in all conscience I cannot advise to use the wheel in a multithread concurrent production environment.

[simcax]

I just hit this snag today, on a Raspberry Pi 4 running Ubuntu. I was wondering if this is nearing production status?

[dvarrazzo]

I might have some resource to make a new psycopg2 release towards the end of the new week.

[simcax]

Sounds awesome 😎 in the meantime, I've started inves igating how to compile it into my image, but that would make our release times longer of course, so would be ecstatic if arm64 support was available. I'll be happy to test it out, in case it's needed with some more tedtcases. I'm using it against Cockroach DB.

[dvarrazzo]

@simcax if you could test the travis setup to build the arm images that would be great. Unfortunately the wheel and travis stack rots pretty quickly and I'm sure surprises will happen. Any help is welcome.

[simcax]

Sure thing. I just need to figure out where to start, as I haven't been involved previously.

[dvarrazzo]

@simcax Packages are built by the https://github.com/psycopg/psycopg2-wheels/ project, pushing the code to github and letting travis.com to pick it up. I think arm builds are disabled there for the problems that travis.org was having.