Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange · 2023-01-20T14:54:17Z

Hoi 👋

linux-aarch64 makes up for almost 10% of all platforms ref giampaolo/psutil#2103

aarch64 has already surpassed windows in terms of downloads for this package. Oracle, Amazon, Google, and Microsoft are all offering aarch64 cloud instances at an undeniable price point compared to amd/intel, so the demand will undoubtedly only grow

this PR is adapted from Add arm64 mac and linux wheels MagicStack/asyncpg#954
uses QEMU emulation for linux arm64 wheels: manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs 😅
manylinux2014 wheels are built with GCC 10, which I think does not guarantee proper functioning of pybind11 (docs).
- so with this PR, linux wheels are built with GCC 12 (manylinux_2_28).
- pip will only install these wheels on linux operating systems with glibc >= 2.28 (mostly all 2020+ linux distributions like debian 10 buster, ubuntu 20.04 focal, almalinux/rhel 8, ...).

the wheels from this PR can be installed with:

# comma separated list for --find-links
export PIP_FIND_LINKS=https://github.com/ddelange/vaex/releases/expanded_assets/core-v4.17.1.post4
pip install --force-reinstall vaex

fixes #2366, fixes #2368, fixes #2397

maartenbreddels · 2023-01-21T08:32:29Z

Hoi 👋

exciting, will take a look early next week!

manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs

that worries me a bit.. :)

groeten,

Maarten

ddelange · 2023-01-21T09:40:18Z

here are all timings: https://github.com/ddelange/vaex/actions/runs/3965720337/usage

depending on how often a month you release vaex, this could eat into the 2k free minutes of GH...

as the parallelization is maximised and they're pushed to PyPI as soon as they're built, most of the wheels will be available soon upon release regardless

here are all the wheels: distributions.zip

ddelange · 2023-01-21T10:32:44Z

interestingly, that was 8260 minutes ^

apparently that's OK? then I don't understand their explanation 🤔 https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes

ddelange · 2023-01-21T10:40:57Z

ah there is a fair amount of duplication in that usage table for whatever reason 🤯

ddelange · 2023-01-22T10:54:32Z

a diff of current PyPI vs the zip above:

 vaex_core-4.16.1-cp310-cp310-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp310-cp310-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-win_amd64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_10_9_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_11_0_arm64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-win_amd64.whl
 vaex_core-4.16.1-cp36-cp36m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp36-cp36m-win_amd64.whl
 vaex_core-4.16.1-cp37-cp37m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp37-cp37m-win_amd64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp38-cp38-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-win_amd64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp39-cp39-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-win_amd64.whl

ddelange · 2023-01-23T07:30:45Z

packages/vaex-core/src/hash_object.cpp

-namespace std {
-    template<>
-    struct hash<PyObject*> {
-        size_t operator()(const PyObject *const &o) const {
-            return PyObject_Hash((PyObject*)o);
-        }
-    };
-}


This was my best guess...

pybind11 ref https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859

failure log https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110

@maartenbreddels any thoughts on this (incl me updating the pybind11 submodule)?

ddelange · 2023-01-23T07:31:30Z

packages/vaex-core/setup.py

@@ -183,12 +183,14 @@ def __str__(self):
      include_package_data=True,
      ext_modules=([extension_vaexfast] if on_rtd else [extension_vaexfast, extension_strings, extension_superutils, extension_superagg]) if not use_skbuild else [],
      zip_safe=False,
+      python_requires=">=3.6",


cibuildwheel parses this to determine which wheels to build

cc @franz101

see also the diff above

ddelange · 2023-02-27T07:19:30Z

I'm guessing this is blocked by #2339

maartenbreddels · 2023-03-06T13:11:53Z

Just letting you know i'm very busy and had a vacation.
Yes, I'll try to get #2339 green first!

ddelange · 2023-05-25T23:20:01Z

fwiw there are now third party free minutes on native arm64 machines, to get rid of the slow qemu builds

maartenbreddels · 2023-07-10T20:25:06Z

Could you try rebasing this?

ddelange · 2023-07-10T20:48:15Z

@maartenbreddels already merged in master 👍

ddelange · 2023-07-11T06:55:46Z

    ERROR: Could not find a version that satisfies the requirement vaex-core<4.17,>=4.17.0 (from vaex)
    ERROR: No matching distribution found for vaex-core<4.17,>=4.17.0

maartenbreddels · 2023-07-11T15:50:10Z

Yeah, a bug/artifact or our release script. Should be good now.

ddelange · 2023-08-03T20:50:04Z

hoi @maartenbreddels 👋

I pulled master and fixed merge conflicts, but it looks like CI is still not very happy. Seeing errors like hdf file missing on disk, and TypeError: train() got an unexpected keyword argument 'early_stopping_rounds'.

Do you think it might be related to this PR?

franz101 · 2023-08-17T20:51:39Z

Just wondering here on the Python packaging. Python 3.6 and 3.7 are now deprecated on the other hand we can bump to 3.10 and 3.11?

to-bee · 2023-08-28T07:48:57Z

Do we have any updates on this MR?

ddelange · 2023-09-01T08:07:30Z

HI @maartenbreddels 👋

Was your s3 account deleted by any chance?

vaex.open('s3://vaex/taxi/yellow_taxi_2009_2015_f32.hdf5?anon=true')

raises

FileNotFoundError: [Errno 2] Path does not exist 'vaex/taxi/yellow_taxi_2009_2015_f32.hdf5'. Detail: [errno 2] No such file or directory

JovanVeljanoski · 2024-02-09T12:32:31Z

Pushed some changes that should fix the failing tests in vaex-ml

maartenbreddels · 2024-02-09T14:18:46Z

Thank you @JovanVeljanoski !
This is starting to look good, I need to fix those files that are missing now, I'm happy to fix that.
The Python 3.6 and 3.7 failures with micromamba I could use some help with.

JovanVeljanoski · 2024-02-09T14:32:12Z

Looks like lightgbm>4. is not available via conda-forge for python < 3.8.
I will attempt to install it via pip to see if that helps.

ddelange · 2024-02-13T17:57:33Z

base_url = 's3://vaex'

    @pytest.mark.slow
    @pytest.mark.parametrize("base_url", ["gs://vaex-data", "s3://vaex"])
    def test_cloud_glob(base_url):
>       assert set(vaex.file.glob(f'{base_url}/testing/*.hdf5', fs_options=fs_options)) >= ({f'{base_url}/testing/xys-masked.hdf5', f'{base_url}/testing/xys.hdf5'})
E       AssertionError: assert set() >= {'s3://vaex/testing/xys-masked.hdf5', 's3://vaex/testing/xys.hdf5'}
E        +  where set() = set([])
E        +    where [] = <function glob at 0x7f8447317f28>('s3://vaex/testing/*.hdf5', fs_options={'anonymous': 'true'})
E        +      where <function glob at 0x7f8447317f28> = <module 'vaex.file' from '/home/runner/work/vaex/vaex/packages/vaex-core/vaex/file/__init__.py'>.glob
E        +        where <module 'vaex.file' from '/home/runner/work/vaex/vaex/packages/vaex-core/vaex/file/__init__.py'> = vaex.file

tests/cloud_dataset_test.py:45: AssertionError

maartenbreddels · 2024-02-14T12:08:32Z

The hash issues are due to dask/dask#10876
I think it's easier to pin dask to <2024.2.0 and keep doing that for a while and see what changes in the future (will they keep changing, or will they revert back to having the same result as before this release).

This version gives different results, although not a problem in production (it will make your cache invalid though), for CI we test that we have stable keys (fingerprints)

maartenbreddels · 2024-02-14T15:51:02Z

Getting greener, but seeing micromamba failing often, and hanging of tests on OSX.

ddelange · 2024-02-22T11:16:00Z

hmm, looks like micromamba is still flakey. maybe relevant? https://stackoverflow.com/a/77333269/5511061

ddelange · 2024-02-23T08:03:57Z

macos seems to be consistently hanging on https://github.com/vaexio/vaex/blob/master/tests/ml/cluster_test.py

any ideas there @JovanVeljanoski?

EwoutH · 2024-02-23T10:22:02Z

Can we make this more manageable by splitting it into multiple smaller PRs? Like:

Removing old Python versions
Updating all the environments and tools used
Adding new Python versions (3.11 and 3.12)
Adding new platforms (arm64)

I feel the size and complication of this PR now holds this effort back.

EwoutH · 2024-03-04T14:05:05Z

With #2417 and #2414 I started with two small steps.

franz101 · 2024-03-08T11:01:52Z

I compiled a list of all stable releases during the time the last build was working:
#2417 (comment)

I'm not sure which package is causing the hanging tests:
I noticed we pinned pytest-async to 0.15 latest is (0.23.5) further catboost maybe needs to be pinned

to-bee · 2024-04-17T09:44:24Z

Hi there. Any plans to release this soonish? Really appreciated!

ddelange · 2024-04-17T10:42:17Z

@to-bee it would be a great help if you can install the wheels (see PR description) and report back your environment info + whether the wheels work in your environment!

to-bee · 2024-04-24T09:25:00Z

@ddelange yes sure.
The wheels are working fine for me. Could install without any problems.
python 3.12.3, Apple M1, ARM64_T6000 arm64, macOS 14.1.1

ddelange mentioned this pull request Jan 21, 2023

Maximize wheel build parallellization ddelange/vaex#1

Closed

ddelange commented Jan 23, 2023

View reviewed changes

ddelange force-pushed the build-matrix branch from beff33a to 7bf26cc Compare February 27, 2023 07:15

This was referenced Jun 30, 2023

[FEATURE-REQUEST] Support Python 3.11.2 #2368

Open

[BUG-REPORT] Failed to build vaex-core (pybind11 issue?) #2366

Open

ddelange changed the title ~~Build aarch64 wheels~~ Build aarch64 wheels and support python 3.11 Jul 10, 2023

maartenbreddels force-pushed the master branch from 8d44942 to d4bf017 Compare July 11, 2023 13:55

CarrieFilion mentioned this pull request Aug 2, 2023

Install with Python 3.11 fails athob/py-ananke#1

Closed

ddelange referenced this pull request in rapidfuzz/RapidFuzz Aug 10, 2023

upgrade cython

dfea552

ddelange mentioned this pull request Aug 30, 2023

Discussion: support bundling libmagic ahupp/python-magic#233

Closed

ddelange force-pushed the build-matrix branch 3 times, most recently from 5680eb9 to 2136629 Compare September 4, 2023 08:28

ci: upgrade xcode for brew install libomp

ba4943b

JovanVeljanoski force-pushed the build-matrix branch from 3adfa7a to ba4943b Compare February 9, 2024 12:27

update requirements-ml to comply with the latest veex-ml expectations.

a346454

try to install lightgbm via pip

d7023a7

fix: only the mini taxi file is on s3 for cost savings

021e60f

maartenbreddels mentioned this pull request Feb 14, 2024

Make tokenization more deterministic dask/dask#10876

Merged

maartenbreddels added 4 commits February 14, 2024 13:09

ci: pin dask<2024.2.0 to get the same hash keys

a44e464

This version gives different results, although not a problem in production (it will make your cache invalid though), for CI we test that we have stable keys (fingerprints)

fix: only the mini taxi file is on s3 for cost savings (2)

915d977

ci: skip notebooks that depend on the dataframe server

658816d

ci: skip ci steps in various old python versions

afa6225

tyapochkin mentioned this pull request Feb 15, 2024

[good first issue - beginner] Add Hamilton example using Vaex DAGWorks-Inc/hamilton#457

Closed

Bump micromamba and other actions

eb2afa3

ci: specific test hangs on ci+osx

1f04e2e

ddelange mentioned this pull request Apr 4, 2024

Parallelize and add more wheel builds alesanfra/dlib-wheels#9

Merged

Bump cibuildwheel, use native arm64 mac runners

167e500

erwanp mentioned this pull request Apr 21, 2024

Remove install dependency + support Python 3.9, 3.10 radis/radis#647

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support CPython 3.11, 3.12, and aarch64 processors #2331

Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange commented Jan 20, 2023 •

edited

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 •

edited

ddelange Feb 27, 2023

ddelange Jan 23, 2023

ddelange Aug 17, 2023

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 •

edited

JovanVeljanoski commented Feb 9, 2024

maartenbreddels commented Feb 9, 2024

JovanVeljanoski commented Feb 9, 2024

ddelange commented Feb 13, 2024

maartenbreddels commented Feb 14, 2024

maartenbreddels commented Feb 14, 2024

ddelange commented Feb 22, 2024

ddelange commented Feb 23, 2024

EwoutH commented Feb 23, 2024 •

edited

EwoutH commented Mar 4, 2024

franz101 commented Mar 8, 2024

to-bee commented Apr 17, 2024

ddelange commented Apr 17, 2024

to-bee commented Apr 24, 2024

Support CPython 3.11, 3.12, and aarch64 processors #2331

Are you sure you want to change the base?

Support CPython 3.11, 3.12, and aarch64 processors #2331

Conversation

ddelange commented Jan 20, 2023 • edited

the wheels from this PR can be installed with:

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 • edited

Choose a reason for hiding this comment

ddelange Feb 27, 2023

Choose a reason for hiding this comment

ddelange Jan 23, 2023

Choose a reason for hiding this comment

ddelange Aug 17, 2023

Choose a reason for hiding this comment

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 • edited

JovanVeljanoski commented Feb 9, 2024

maartenbreddels commented Feb 9, 2024

JovanVeljanoski commented Feb 9, 2024

ddelange commented Feb 13, 2024

maartenbreddels commented Feb 14, 2024

maartenbreddels commented Feb 14, 2024

ddelange commented Feb 22, 2024

ddelange commented Feb 23, 2024

EwoutH commented Feb 23, 2024 • edited

EwoutH commented Mar 4, 2024

franz101 commented Mar 8, 2024

to-bee commented Apr 17, 2024

ddelange commented Apr 17, 2024

to-bee commented Apr 24, 2024

ddelange commented Jan 20, 2023 •

edited

ddelange commented Jan 21, 2023 •

edited

ddelange commented Jan 21, 2023 •

edited

ddelange Jan 23, 2023 •

edited

ddelange commented Sep 1, 2023 •

edited

EwoutH commented Feb 23, 2024 •

edited