python wheels using rustc #5435

th0ma7 · 2022-09-20T00:24:27Z

Description

python wheels using rustc - Using another approach from #5399

Extras:

Update python310 to version 3.10.7 (possibly 3.10.8? https://peps.python.org/pep-0619/)
Update duplicity to version 0.8.23
Fix mysql-connector-c which now fails to build
As needing to update duplicity, shoot for version 1.0.1 (tested on x64-7.0, armv7-7.0, armv5-6.1)

TODO: none left

Fixes #

Checklist

Build rule all-supported completed successfully
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

th0ma7 · 2022-09-20T01:12:07Z

@publicarray reading back on one of your comments #4971 (comment) : Any changes to the Dockerfile won't actually be ran unless merged to master?

Safihre · 2022-09-21T08:05:22Z

@publicarray reading back on one of your comments #4971 (comment) : Any changes to the Dockerfile won't actually be ran unless merged to master?

Yes, but it's also only build when it's on master, so it's not really tested.

publicarray · 2022-09-21T08:14:07Z

@publicarray reading back on one of your comments #4971 (comment) : Any changes to the Dockerfile won't actually be ran unless merged to master?

Yes, but it's also only build when it's on master, so it's not really tested.

Yea I suppose the idea here was that you do build the docker container locally first and verify it before pushing the changes to master. However you can create a docker build in your fork if that is easier. yea it's not a good idea to do docker builds in PRs as we don't want to modify the published image until the changes land in master.

miigotu · 2022-09-23T19:09:03Z

Is the rust env getting sourced?

This is how I do it in our docker file:
https://github.com/SickChill/SickChill/blob/master/contrib/docker/Dockerfile.rpi#L37

Dockerfile

mk/spksrc.tc.mk

miigotu · 2022-09-29T19:49:44Z

So, I've been trying to get 32 bit arms builds working using docker buildx, and came across the same issue. There is a readdir bug in qemu for 32 bit arm that makes it's way all the way into the crossenv. This link is a docker build but I think the solution might be a similar problem in this cross toolchain. Also, rustup can't tell the architecture with an 32 bit arm architecture, so it can never find the correct compiler for arm6 and 7 and sends the objects to the wrong linker for arm 5/6

The fix for this is a mixture of the solution in the link here, and a patch to rustup I had done awhile back (I'll post that when I update the patch and test these fixes in https://github.com/miigotu/python-docker-images

matrix-org/synapse#9403 (comment)

th0ma7 · 2022-09-29T21:16:30Z

So, I've been trying to get 32 bit arms builds working using docker buildx, and came across the same issue. There is a readdir bug in qemu for 32 bit arm that makes it's way all the way into the crossenv. This link is a docker build but I think the solution might be a similar problem in this cross toolchain. Also, rustup can't tell the architecture with an 32 bit arm architecture, so it can never find the correct compiler for arm6 and 7 and sends the objects to the wrong linker for arm 5/6

That's quite an interesting catch, although, the issue is also hapening on other archs and not only armv7. I can reproduce a rather similar output using i686 (arch-evansport-7.0) and aarch64 for instance. All leading to a linking time issue.

miigotu · 2022-09-30T00:02:28Z

Yeah I have the same issue with 386/686. I think it is related.

That's part of the patch for rustup is to get it to recognize the bitness as 32, it detects a 32 bit arch at 64 bit currently. I'll go quick and see if I can find my bug report from last year.

miigotu · 2022-09-30T00:07:44Z

rust-lang/rustup#2700

miigotu · 2022-09-30T06:16:27Z

Man this is super annoying, this works locally to build armv7, but failed in github actions:
https://github.com/miigotu/python-docker-images/actions/runs/3156651360/workflow

from this dockerfile: https://github.com/miigotu/python-docker-images/blob/main/python/python-cryptography.dockerfile

docker buildx build --platform linux/arm/v7 -f python/python-cryptography.dockerfile . --progress plain --ulimit nofile=262144:262144 --output=type=local,dest=/tmp/cryptography-wheels

th0ma7 · 2022-09-30T12:33:12Z

mk/spksrc.cross-rust.mk

-# Use distrib folder as cargo download cache
-CARGO_HOME_PATH=/spksrc/distrib/cargo
-ENV += CARGO_HOME=$(CARGO_HOME_PATH)
-ENV += PATH=:$(CARGO_HOME_PATH)/bin/:$(PATH)


~~Indeed but as the CARGO_HOME is being set to /usr/local/ therefore /usr/local/bin is automagically being in the default PATH.~~

This was reworked and now sits under /opt where cargo & rustup have independant directories. Theses values have been added in the default wide environment through spksrc.tc.mk and the /opt/cargo/bin to spksrc.common.mk

miigotu · 2022-09-30T18:16:02Z

Same dockerfile works locally for all of them smh
docker buildx build --platform linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -f python/python-cryptography.dockerfile . --progress plain --ulimit nofile=262144:262144 --output=type=local,dest=wheels

cryptography-wheels.zip

th0ma7 · 2022-09-30T20:29:52Z

First thing to notice in here, the example you're refering to is using venv instead of crossenv. Something I'm not sure I understand (still ramping-up on docker), is it starting a docker in qemu where host & target are the same? Or is it cross-compiling from a x86_64 host?

Anyhow, having a few pending patches on my end and trying to build using spksrc my "updated" docker environment where the host is x86_64.

My first thing to try is, is it even cross-compiling using rust? Lets take ripgrep as an easy app to build:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

It works! Now let's try that using CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu instead of enforcing --target:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Still working! So the theory is that I have a working rustc along with a proper recognition of my target gcc as default linker and CARGO* and RUSTUP* environment variables are also recognized and functional.

Now, following your Dockerfile example, along with using host default python3, still on x86_64 host:

# python3 --version
Python 3.9.2
# apt-get update
# apt-get install python3-venv
# VENV_PATH=/opt/venv python3 -m venv $VENV_PATH --upgrade
# VENV_PATH=/opt/venv python3 -m pip install --upgrade setuptools pip
# VENV_PATH=/opt/venv python3 -m pip install --upgrade setuptools-rust wheel
# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ VENV_PATH=/opt/venv python3 setup.py bdist_wheel --py-limited-api=cp36

Boom! working! But hold-on, why is the wheel extension being x86_64?

# ll dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl
-rw-r--r-- 1 root root 1214001 Sep 30 19:46 dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl

After investigating a little further, looks like everything in the wheel file is ?OK?:

# file ./hazmat/bindings/_rust.abi3.so
./hazmat/bindings/_rust.abi3.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

Now let's try out the exact same thing, this time using our host native built python3.10:

# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m venv $VENV_PATH --upgrade
# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m pip install --upgrade setuptools pip
# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m pip install --upgrade setuptools-rust wheel

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 setup.py bdist_wheel --py-limited-api=cp36

Again, boom! Although it works with a few deprecation warnings:

build/temp.linux-x86_64-cpython-310/cryptography.hazmat.bindings._openssl.c: In function ‘_cffi_d_ASN1_STRING_data’:
build/temp.linux-x86_64-cpython-310/cryptography.hazmat.bindings._openssl.c:7957:3: warning: ‘ASN1_STRING_data’ is deprecated [-Wdeprecated-declarations

Let's investigate further:

# ll dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl
-rw-r--r-- 1 root root 2040082 Sep 30 20:10 dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl

After unziping:

# find . -name "*.so"
./cryptography/hazmat/bindings/_openssl.abi3.so
./cryptography/hazmat/bindings/_rust.abi3.so
# file ./cryptography/hazmat/bindings/_openssl.abi3.so ./cryptography/hazmat/bindings/_rust.abi3.so
./cryptography/hazmat/bindings/_openssl.abi3.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a1db744141c71550a25dac57f63994c81be4cf90, with debug_info, not stripped
./cryptography/hazmat/bindings/_rust.abi3.so:    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

So clearly a mismatch here. But it did built sort of "succesfully" using a venv. But clearly this wheel output is useless as having arch mismatchs at every levels. But it did demonstrated that it can build, which is already a start.

Now, challenge really is, building using a crossenv ...

miigotu · 2022-09-30T22:38:59Z

First thing to notice in here, the example you're refering to is using venv instead of crossenv. Something I'm not sure I understand (still ramping-up on docker), is it starting a docker in qemu where host & target are the same? Or is it cross-compiling from a x86_64 host?

Anyhow, having a few pending patches on my end and trying to build using spksrc my "updated" docker environment where the host is x86_64.

My first thing to try is, is it even cross-compiling using rust? Lets take ripgrep as an easy app to build:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

It works! Now let's try that using CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu instead of enforcing --target:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Still working! So the theory is that I have a working rustc along with a proper recognition of my target gcc as default linker and CARGO* and RUSTUP* environment variables are also recognized and functional.

Now, following your Dockerfile example, along with using host default python3, still on x86_64 host:

# python3 --version
Python 3.9.2
# apt-get update
# apt-get install python3-venv
# VENV_PATH=/opt/venv python3 -m venv $VENV_PATH --upgrade
# VENV_PATH=/opt/venv python3 -m pip install --upgrade setuptools pip
# VENV_PATH=/opt/venv python3 -m pip install --upgrade setuptools-rust wheel
# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ VENV_PATH=/opt/venv python3 setup.py bdist_wheel --py-limited-api=cp36

Boom! working! But hold-on, why is the wheel extension being x86_64?

# ll dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl
-rw-r--r-- 1 root root 1214001 Sep 30 19:46 dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl

After investigating a little further, looks like everything in the wheel file is ?OK?:

# file ./hazmat/bindings/_rust.abi3.so
./hazmat/bindings/_rust.abi3.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

Now let's try out the exact same thing, this time using our host native built python3.10:

# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m venv $VENV_PATH --upgrade
# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m pip install --upgrade setuptools pip
# VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 -m pip install --upgrade setuptools-rust wheel

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ VENV_PATH=/opt/venv /spksrc/native/python310/work-native/install/usr/local/bin/python3 setup.py bdist_wheel --py-limited-api=cp36

Again, boom! Although it works with a few deprecation warnings:

build/temp.linux-x86_64-cpython-310/cryptography.hazmat.bindings._openssl.c: In function ‘_cffi_d_ASN1_STRING_data’:
build/temp.linux-x86_64-cpython-310/cryptography.hazmat.bindings._openssl.c:7957:3: warning: ‘ASN1_STRING_data’ is deprecated [-Wdeprecated-declarations

Let's investigate further:

# ll dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl
-rw-r--r-- 1 root root 2040082 Sep 30 20:10 dist/cryptography-38.0.1-cp36-abi3-linux_x86_64.whl

After unziping:

# find . -name "*.so"
./cryptography/hazmat/bindings/_openssl.abi3.so
./cryptography/hazmat/bindings/_rust.abi3.so
# file ./cryptography/hazmat/bindings/_openssl.abi3.so ./cryptography/hazmat/bindings/_rust.abi3.so
./cryptography/hazmat/bindings/_openssl.abi3.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a1db744141c71550a25dac57f63994c81be4cf90, with debug_info, not stripped
./cryptography/hazmat/bindings/_rust.abi3.so:    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

So clearly a mismatch here. But it did built sort of "succesfully" using a venv. But clearly this wheel output is useless as having arch mismatchs at every levels. But it did demonstrated that it can build, which is already a start.

Now, challenge really is, building using a crossenv ...

I'm not building these with spksrc at all, because it isn't necessary. All pyo3 wheels built on a system with libc (non-alpine) will run on the same architecture they were built for, regardless of the host.

Buildx is a crossenv. The GitHub actions runs in an x86_64 host container always (ubuntu-latest is Ubuntu:jammy right now I believe). Buildx runs each build in a qemu cross ENV either determined by the --platform argument or possibly overridden on the FROM line of the dockerfile.

Venv is a totally different thing, that just isolates the python environment from the system python environment, so there are no conflicts with the system python packages or contamination of the build/install of what you are building. That could be left out entirely from the dockerfile and it would work, it just isn't best practices.

So essentially, if this build worked in GitHub actions, very easily we could build every cross wheel in GitHub actions directly and then pass them into the spksrc build and build spksrc without needing rust at all.

It was just trying to prove that it isn't spksrc toolchain that is the problem, there is something wonky with GitHub actions that's making it not possible to build 32 bit builds (since I can build the same action locally but it fails to find rustc on actions). I'm going to break it back into cargo fetch on amd64 and then build on the --platform=$TARGETPLATFORM stage and see if that fixes it on GitHub actions.

If I get this built, I will create releases containing all of the built wheels, and you can reference them with --find-links and always avoid compile errors (and reduce storage usage and build time) on spksrc.

miigotu · 2022-09-30T22:40:09Z

I'll add the build log later, it absolutely is cross compiling the rust and c extensions.

miigotu · 2022-10-01T01:51:36Z

Once cryptography 35.0.0+ is built, you could install poetry right to the crossenv and that would be great.

It will also require tomli up until python 3.11, where tomllib has been added to python stdlib https://docs.python.org/3.11/library/tomllib.html

I'm not sure of flit dependencies, but that is the other major tool used for managing dependencies with pyproject.toml. I think it needs either toml or toml-w

Dockerfile

th0ma7 · 2022-10-01T02:37:04Z

I gave it a shot on both Docker and a native VM to see if this isn't something in the host detection or similar... Sadly same result.

It's really easy to trigger with this PR, once the build is complete (e.g. make arch-armv7-7.0) do the following. First setup your PATH to point towards the crossenv first:

$ export PATH=/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:$PATH

Then build:

$ CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

Interestingly enough it gets a little farther using RUSTFLAGS such as:

RUSTFLAGS="--target=aarch64-unknown-linux-gnu -Clinker=/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

A call using crossenv pip will also result with the same outcome:

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
pip install cryptography==38.0.1

miigotu · 2022-10-01T02:52:33Z

It's just crazy that I can build them with buildx locally but not on GitHub, with the same sources, commands, environment, and everything.

I'm going to try another method to compile right in the actions, rather than inside a dockerfile.

I can set up python in the action, set up rust toolchain (with multiple targets), and build/compile with setuptools-rust.

Now whether that gets around the bug in qemu (or GitHub, I'm not 100% sure the culprit yet), I don't know.

I suspect it could be because the action runner itself is in a container, and then we run another container inside that. Which I expect is why when I run the buildx command on my host machine it works, but on GitHub it doesn't. There's really no other difference between how I run here and there.

miigotu · 2022-10-01T03:01:31Z

I imagine my next attempt to look something like this:

actions/setup-python (matrix with python 3.7-3.11-dev)
Then something similar to this:
https://github.com/actions-rs/example/blob/master/.github/workflows/cross_compile.yml
Then-
run: |
pip download --no-deps :all: cryptography
unzip
cd and python setup.py bdist_wheel --py-limited-abi=cp36 cryptography

miigotu · 2022-10-01T03:31:57Z

@th0ma7 seems someone has already done what I was thinking, check this out: https://github.com/PyO3/setuptools-rust/pull/185/files

th0ma7 · 2022-10-01T11:04:17Z

I imagine my next attempt to look something like this:

actions/setup-python (matrix with python 3.7-3.11-dev) Then something similar to this: https://github.com/actions-rs/example/blob/master/.github/workflows/cross_compile.yml Then- run: | pip download --no-deps :all: cryptography unzip cd and python setup.py bdist_wheel --py-limited-abi=cp36 cryptography

You can already do something close to this by using cross/cryptography which in turns doesn't use pip for building. You simply have to add it to DEPENDS variable of spk/python310 for a start. Although the github workflow won't be the same.

messense · 2022-10-01T11:57:41Z

Cross-post from PyO3/setuptools-rust#294

This circles back to benfogle/crossenv#65, after removing LIBRARY_PATH env var from /spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin/python and adding PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/crossenv/lib env var for pyo3, the build succeeded for me locally.

th0ma7 · 2022-10-01T13:05:33Z

Good news! I got it working PyO3/setuptools-rust#294 (comment)

I have to think about the various code changes I need to do to have a working fix (or pending a new setuptools-rust release) but it does work both using cross and pip from a crossenv... there is light at the end of the tunnel finally!

miigotu · 2022-10-01T23:33:50Z

Awesome, I hope that same thing fixes a straight build in gh actions too. Maybe this fix will bring back prebuilt wheels for arm32 and ppc64le

th0ma7 · 2022-10-02T15:32:02Z

@hgy59 got a working PR to generate rust-based wheels. I still have a few details to polish but overall code is looking good.

The real thing in here it the updated docker image. I propose pushing it into a different PR which in turn would allow me to fully test things in this one in particular (as I can't using github-action until it's merged to master, although I tested it manually).

Changes to mention related to the Docker image:

Bumped to Debian 11 (e.g. stable)
Installation by default of cargo/rustc environment

I did a lot of build testing and overall things is looking good. Although I'm sure there will be things to adjust from migrating to Debian 11 resulting in a few bumps down the road that I'm confident we can address afterwards.

Thoughts?

th0ma7 · 2022-10-09T15:31:24Z

@publicarray examining the build errors it does use the latest Docker image. There's just a few details pending that I'm now fixing within the python310 build itself. Otherwise all is good, thnx.

cross/cryptography-legacy/Makefile

miigotu · 2022-10-10T04:57:17Z

Build passed! Great work!

This piece of code is so unmaintained that it requires an old legacy version of cmake to run. Latest changes allow using Debian 10 "Buster" of cmake and thus makes it build again.

* cross-rust.mk: Fix regular build post PR #5435 * finalize rust fix - remove unused code Co-authored-by: hgy59 <hpgy59@gmail.com>

th0ma7 mentioned this pull request Sep 20, 2022

Error when cross-compiling pyca/cryptography#7629

Closed

miigotu reviewed Sep 23, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

th0ma7 mentioned this pull request Sep 28, 2022

setuptools-rust fails to cross-compile cryptography using crossenv PyO3/setuptools-rust#294

Closed

messense reviewed Sep 29, 2022

View reviewed changes

mk/spksrc.tc.mk Outdated Show resolved Hide resolved

messense reviewed Sep 29, 2022

View reviewed changes

mk/spksrc.tc.mk Show resolved Hide resolved

th0ma7 commented Sep 30, 2022

View reviewed changes

miigotu reviewed Oct 1, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

python310: Re-enable cross/cryptography to use x64 build fix

164d194

th0ma7 added 2 commits October 9, 2022 17:45

python310: Add comments for numpy & pycryptodome(x)

e85853f

python310: Fix += for various variables due to many wheels

993802b

miigotu reviewed Oct 9, 2022

View reviewed changes

cross/cryptography-legacy/Makefile Show resolved Hide resolved

python310: psutil 5.9.1 is last known to build on armv5/ppc

bcd3400

th0ma7 added 15 commits October 10, 2022 13:31

python310: Testing all wheels done, disabling except cryptography

dd2f685

python310: Update from v3.10.6 to v3.10.7

09cdc1c

native/cmake: Update to latest stable

e89377c

native/cmake-legacy: Provide a Debian 10 "Buster" version of cmake

f19eff6

cmake.mk: Add CMAKE_LEGACY and fix exported PATH

72a3fe9

mysql-connector-c: Make usage of USE_NATIVE_CMAKE_LEGACY

a0c6d52

This piece of code is so unmaintained that it requires an old legacy version of cmake to run. Latest changes allow using Debian 10 "Buster" of cmake and thus makes it build again.

cmake.mk: For CMAKE_PATH to work from spk need to use $(WORK_DIR)

1a97424

python310: Enable mysqlclient wheel build using latest changes

4d18244

python310: Disable all optional wheels in perapration for release

e30710a

python310: Update from version 3.10.7 to 3.10.8

b97a682

duplicity: Update from version 0.8.12 to 1.0.1

7b9f9f2

bcrypt >= 4.0 also requires PYO3_CROSS* variables

682807f

duplicity: psutil==5.9.1 is last know working version on armv5/ppc

ca8bef3

cross-cmake-env.mk: Fix typo and use $(WORK_DIR) for nasm def.

fe6bf2e

cross-cmake-env.mk: Forgot to re-export nasm using ENV variable

2dbf000

th0ma7 merged commit acd9a36 into SynoCommunity:master Oct 12, 2022

th0ma7 deleted the rustc-wheels branch October 12, 2022 23:19

th0ma7 mentioned this pull request Oct 12, 2022

META: Python 3.10 / 3.11 package status #5043

Open

10 tasks

th0ma7 removed the status/needs-review label Oct 22, 2022

th0ma7 added a commit to th0ma7/spksrc that referenced this pull request Oct 24, 2022

cross-rust.mk: Fix regular build post PR SynoCommunity#5435

cb4260c

th0ma7 mentioned this pull request Oct 24, 2022

cross-rust.mk: Fix regular build post PR #5435 #5466

Merged

10 tasks

th0ma7 added a commit that referenced this pull request Oct 28, 2022

cross-rust.mk: Fix regular build post PR #5435 (#5466)

aa46ebc

* cross-rust.mk: Fix regular build post PR #5435 * finalize rust fix - remove unused code Co-authored-by: hgy59 <hpgy59@gmail.com>

hgy59 mentioned this pull request Dec 23, 2022

Add matrix-synapse #5399

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python wheels using rustc #5435

python wheels using rustc #5435

th0ma7 commented Sep 20, 2022 •

edited

Loading

th0ma7 commented Sep 20, 2022

Safihre commented Sep 21, 2022

publicarray commented Sep 21, 2022

miigotu commented Sep 23, 2022 •

edited

Loading

miigotu commented Sep 29, 2022 •

edited

Loading

th0ma7 commented Sep 29, 2022

miigotu commented Sep 30, 2022

miigotu commented Sep 30, 2022

miigotu commented Sep 30, 2022 •

edited

Loading

th0ma7 Sep 30, 2022 •

edited

Loading

miigotu commented Sep 30, 2022

th0ma7 commented Sep 30, 2022

miigotu commented Sep 30, 2022 •

edited

Loading

miigotu commented Sep 30, 2022 •

edited

Loading

miigotu commented Oct 1, 2022

th0ma7 commented Oct 1, 2022 •

edited

Loading

miigotu commented Oct 1, 2022

miigotu commented Oct 1, 2022 •

edited

Loading

miigotu commented Oct 1, 2022

th0ma7 commented Oct 1, 2022

messense commented Oct 1, 2022

th0ma7 commented Oct 1, 2022

miigotu commented Oct 1, 2022

th0ma7 commented Oct 2, 2022

th0ma7 commented Oct 9, 2022

miigotu commented Oct 10, 2022

python wheels using rustc #5435

python wheels using rustc #5435

Conversation

th0ma7 commented Sep 20, 2022 • edited Loading

Description

Checklist

Type of change

th0ma7 commented Sep 20, 2022

Safihre commented Sep 21, 2022

publicarray commented Sep 21, 2022

miigotu commented Sep 23, 2022 • edited Loading

miigotu commented Sep 29, 2022 • edited Loading

th0ma7 commented Sep 29, 2022

miigotu commented Sep 30, 2022

miigotu commented Sep 30, 2022

miigotu commented Sep 30, 2022 • edited Loading

th0ma7 Sep 30, 2022 • edited Loading

Choose a reason for hiding this comment

miigotu commented Sep 30, 2022

th0ma7 commented Sep 30, 2022

miigotu commented Sep 30, 2022 • edited Loading

miigotu commented Sep 30, 2022 • edited Loading

miigotu commented Oct 1, 2022

th0ma7 commented Oct 1, 2022 • edited Loading

miigotu commented Oct 1, 2022

miigotu commented Oct 1, 2022 • edited Loading

miigotu commented Oct 1, 2022

th0ma7 commented Oct 1, 2022

messense commented Oct 1, 2022

th0ma7 commented Oct 1, 2022

miigotu commented Oct 1, 2022

th0ma7 commented Oct 2, 2022

th0ma7 commented Oct 9, 2022

miigotu commented Oct 10, 2022

th0ma7 commented Sep 20, 2022 •

edited

Loading

miigotu commented Sep 23, 2022 •

edited

Loading

miigotu commented Sep 29, 2022 •

edited

Loading

miigotu commented Sep 30, 2022 •

edited

Loading

th0ma7 Sep 30, 2022 •

edited

Loading

miigotu commented Sep 30, 2022 •

edited

Loading

miigotu commented Sep 30, 2022 •

edited

Loading

th0ma7 commented Oct 1, 2022 •

edited

Loading

miigotu commented Oct 1, 2022 •

edited

Loading