Provide pre-compiled wheels for Mac OS 12 (Monterey)

[RootLUG]

Hello,
as the name title suggest, this issue is about providing precompiled wheels for the newest MacOS Monterey (12.0).
The main psycopg package installs fine (via poetry or pip! but the provider (psycopg-binary) can't be installed. Pip currently doesn't find the matching wheel/distribution for the latest MacOS, I would guess the OS selector does not match the existing uploaded wheels.
Tested on py3.9 with poetry & pip, Mac OS 12.0.1; I've checked also wheels on pypi and it appears this is the case for all other python version for which the psycopg-binary is released as well.

[dvarrazzo]

Does cibuildwheel enable to create and test such packages on GitHub Actions?

[RootLUG]

I don't believe so. In CI I see it builds and test the wheel for Mac OS py3.9 which matches my environment but the wheel name (pip filters on filename) that is build contains also the macos version (psycopg_binary-3.0.4-cp39-cp39-macosx_10_9_x86_64.whl).

Here are some relevant logs I found in buildci:

Build options:
  platform: 'macos'
  architectures: {<Architecture.universal2: 'universal2'>, <Architecture.x86_64: 'x86_64'>, <Architecture.arm64: 'arm64'>}
  build_selector: BuildSelector(build_config='cp39-macosx_x86_64')

...

Building cp39-macosx_x86_64 wheel
CPython 3.9 macOS x86_64

...

Building wheels for collected packages: psycopg-binary
    Building wheel for psycopg-binary (PEP 517): started
    Building wheel for psycopg-binary (PEP 517): finished with status 'done'
    Created wheel for psycopg-binary: filename=psycopg_binary-3.0.4-cp39-cp39-macosx_10_9_x86_64.whl size=527183 sha256=46ab30feba31e3b9ac300af5f91fdf53a497d4ee074044e1651435890a5cf1ba
    Stored in directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-ephem-wheel-cache-5uwk8le0/wheels/29/20/88/f5f60833d067ccc96553500a229cd797858bab78af94536410
  Successfully built psycopg-binary

I see the buildselector does not contain os version selector (cp39-macosx_x86_64) but the resulting wheel does, If the wheel is independent of the mac os version maybe that can be tweaked when building so it can be used in other versions besides 10.9? I am not familiar with cibuildwheel so I'm just guessing here

[RootLUG]

I see in the buildci docs it supports the following macos platform identifiers:
cp39-macosx_x86_64
cp39-macosx_universal2
cp39-macosx_arm64

Currently the "cp39-macosx_x86_64" is used which may be the issue as that is for older intel based macs, the new m1 are arm64 based and apple made also "universal binary" format which I think is the "cp39-macosx_universal2" to run on both x64 and arm macs so enabling/building for that one may increase the compatibility for different macs

[dvarrazzo]

We can build macos2_universal packages, but we cannot test them with the hardware available, which for me is a no deal.

We need someone to provide psycopg with some M1 runner for GitHub Actions.

[RootLUG]

I think the universal package can use the existing runner (non-m1 hardware runner) as it's intended/designed to be a format that works on both intel and m1 without any changes so in theory if it builds and tests fine in the current ci mac runner it should work on m1 as well (the same wheel file). If it helps I can then also test it on my M1 and also intel based Mac to confirm the compatibility.

[dvarrazzo]

I like the theory: it works, in theory. But I also like the practice, that, in practice, works.

Ok, I'm going to bite the bullet on this. If you can provide a MR to fix the packages.yaml and build universal packages I will stop caring and release them, after some one-off manual testing.

[RootLUG]

So I played with this in a forked repo and it's probably a no go.
The "universal2" libraries do not produce correct wheels as the wheels link to libraries outside the wheel and the delocate is not able to fix it automatically and fails.
The arm64 do compile on intel runners but after installing the package and importing psycopg I get the following error (I also have the libpq library installed):

>>> import psycopg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/intense/Workspace/aura/venv/lib/python3.9/site-packages/psycopg/__init__.py", line 9, in <module>
    from . import pq  # noqa: F401 import early to stabilize side effects
  File "/Users/intense/Workspace/aura/venv/lib/python3.9/site-packages/psycopg/pq/__init__.py", line 114, in <module>
    import_from_libpq()
  File "/Users/intense/Workspace/aura/venv/lib/python3.9/site-packages/psycopg/pq/__init__.py", line 106, in import_from_libpq
    raise ImportError(
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: dlopen(/Users/intense/Workspace/aura/venv/lib/python3.9/site-packages/psycopg_binary/pq.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_PQbackendPID'
- couldn't import psycopg 'python' implementation: libpq library not found

I would say then it would be safer to wait till M1/arm64 runners are natively available in github actions so it can be build and tested on native hardware. Here are the logs from the build packages action for those who are interested: https://github.com/RootLUG/psycopg/actions/runs/1494744423

[hanssens]

FWIW => Support for M1/ARM64 with GitHub Actions can be tracked in issue:

• Support for VMs on Apple M1 actions/runner-images#2187

[sameerlumapps]

running into the same issue as OP. Wish there is a fix soon for this.

[palminha]

Github Actions support for M1 is already in the roadmap... github/roadmap#528

[dvarrazzo]

I've built some test packages for Apple M1.

Please try to install them with:

pip3 install -i https://test.pypi.org/simple/ "psycopg[binary]"

and verify that they work with something like:

python3
>>> import psycopg
>>> psycopg.__version__
'3.0.15'
>>> psycopg.pq.__impl__
'binary'
>>> psycopg.connect("dbname=postgres").execute("SELECT 1+1").fetchone()[0]
2

[hyzyla]

➜  ~/Developer/test-psycopg python3 --version
Python 3.10.4
➜  ~/Developer/test-psycopg python3 -m venv venv
source %                                                                                                                                                                            ➜  ~/Developer/test-psycopg source venv/bin/activate
(venv) ➜  ~/Developer/test-psycopg pip3 install -i https://test.pypi.org/simple/ "psycopg[binary]"
Looking in indexes: https://test.pypi.org/simple/
Collecting psycopg[binary]
  Downloading https://test-files.pythonhosted.org/packages/9b/95/0ee469cb2756ba1b32f10d6fe50914ddc859b3c617ca42e53fa28f78373a/psycopg-3.0.15-py3-none-any.whl (144 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.1/144.1 KB 1.3 MB/s eta 0:00:00
Collecting psycopg-binary==3.0.15
  Downloading https://test-files.pythonhosted.org/packages/d9/22/61709b48f1adff6bd9aefb13252f9b9061a9a0613d60029f0d084f726763/psycopg_binary-3.0.15-cp310-cp310-macosx_11_0_arm64.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 2.8 MB/s eta 0:00:00
Installing collected packages: psycopg-binary, psycopg
Successfully installed psycopg-3.0.15 psycopg-binary-3.0.15
(venv) ➜  ~/Developer/test-psycopg python 
Python 3.10.4 (main, Jul 10 2022, 16:57:06) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
>>> psycopg.__version__
'3.0.15'
>>> psycopg.pq.__impl__
'binary'
>>> psycopg.connect("postgres://reader@localhost:5432/pfmegrnargs").execute("SELECT 1+1").fetchone()[0]
2
>>> 

works fine

[hyzyla]

Is psycopg2 will receive binary wheels also? 🥺