CI: Cross compile wheel for armv7l (Raspberry Pi 4) using Github Actions CI

[Lauszus]

Fixes #6567, fixes #6413, fixes #6474, fixes #6487

[matham]

Looks great, but I have two questions.

• How reliable are these cross builds?
• How do you designate these wheels to a specific pi version? E.g. I'm guessing this would only be usable on the Pi4? But then the name wouldn't indicate this to pip I'm guessing as unlike manylinux there's no standard!?

[Lauszus]

How reliable are these cross builds?

I have only tested it quickly on a Raspberry Pi 4 at work. I'll do some more tests the coming days.

How do you designate these wheels to a specific pi version? E.g. I'm guessing this would only be usable on the Pi4? But then the name wouldn't indicate this to pip I'm guessing as unlike manylinux there's no standard!?

Hmm I am actually not sure if they will work with the older Raspberry Pis. It could be the case. I'll test that tomorrow.

I believe it should work with any armv7l based board if they have compatible OpenGL drivers.

Yes it's not a standard. Worst case I'll just create a separate repo and publish them there for every Kivy release.

Btw any idea why the Travis build is not triggering?

[matham]

Not sure why travis is not triggered. Feel free to fix the code to run with the other linux wheel building and I'll merge it and we'll see how it works.

I'm assuming you tested it locally?

[Lauszus]

@matham I got the Travis build working, but it's exceeding the maximum time limit of 50 minutes: https://travis-ci.org/kivy/kivy/jobs/603837010 :/

Yes I used an instance on AWS to build it and then installed and tested the wheel on a Raspberry Pi 4 locally.

[Lauszus]

@matham I got it working by switching to Github Actions CI, as the cross compiling took 1 hr and 15 min and the maximum time for a Travis build is 50 min in the free version.

I managed to speed up the build a lot by creating a base image with all the apt-get dependencies: https://hub.docker.com/r/lauszus/kivy-armv7l-base (source: https://github.com/Lauszus/kivy-armv7l-base) and adding piwheels as an extra index, so it does not have to compile Cython as well: 038b94f

Since I am not a contributor I can not enable Github Actions for this repo: https://github.com/kivy/kivy/actions, but you should be able to trigger the build once this PR is merged.

You can see a successful build at my fork: https://github.com/Lauszus/kivy/commit/6e9b6bda01d91d1db8e7f1638106fd9f64782bda/checks?check_suite_id=285162135 and here is the link to the build wheel: https://github.com/Lauszus/kivy/suites/285162135/artifacts/197175 for anyone that wants to test it. The wheel is automatically uploaded as an artifact every time the CI job is run, so this also allows people to quickly test the master branch without having to recompile Kivy.

I also enabled so the wheel will automatically get uploaded to the release page when a new tag is created.

[Lauszus]

@matham anything blocking this from being merged? :)

[matham]

Sorry for not getting back to you earlier - I have very little time these days for Kivy, and wasn't sure exactly how to handle this because of the github action switch.

Github actions

First gh actions was only publicly enabled recently so we couldn't have merged this before that anyway. But, also this is entirely different from our current workflow and the team would need to discuss many of these changes before we can merge it.

First, we'd need to agree with switching to gh actions. I expect that will happen eventually, but cannot be sure until the team actually agrees. But even if/when we do switch, the specific workflow you have chosen is very different than the current one. E.g. we only generate wheels once a day rather than for each commit - because it takes so much longer to generate them. We don't upload directly to the release page on GH, but rather to our server.

I except we'll take the time to re-write the full workflow then and am wary of changing it up until then.

wheels

Regarding the actual wheels, I didn't have the time to test the wheel and would want some confirmation from multiple people that it works on a clean Pi and on which py versions it works. Because otherwise, we'll get a bunch of bug reports and we don't have the resources really to fix it atm.

That's not to say I think we shouldn't generate these wheels somehow, just that I want more confirmation that it works. After that, and when we figure out the CI thing, we could add a link on the rpi page to the wheel on the server.

If you get it to work on travis, I would probably merge it because then we can just add it to the server and people can test it etc without major changes to the workflow. Otherwise, I'm not sure we can merge it yet.

[matham]

So, we recently switched to using github actions. Please take a look at the workflows structure and update this PR similarly, if you have time.

There's one request I'd mention, which I didn't bring up before - to not use a docker image which is not under kivy's control. If we need to use a pre-compiled image beyond the standard OSs, it would need to be published by Kivy.

[matham]

In addition, support for armv7l will be in the new manylinux2014 (https://www.python.org/dev/peps/pep-0599/#the-manylinux2014-policy, pypa/manylinux#338), but that won't be ready for a bit.

[Lauszus]

@matham okay, I'll update the PR when I have time.

Yes I understand you point about the Docker image. I will simply remove it causing the build to take longer time, but since we are now using Github CI we will not be restricted by the 50 min build time of Travis.

[Lauszus]

@matham I have now updated the PR, so now the yaml file is basically identical to the one for manylinux :)

One thing we could improve would be to compile it for several version of Python. Currently I only compile it for Python 3.7.

I also added the installation instructions for the Raspberry Pi 4, which should then close a bunch of issues: #6413, #6474 and #6487.

For anyone that is interested, then the cross compiled wheel can be downloaded here: https://github.com/kivy/kivy/suites/364981942/artifacts/751103.

[Lauszus]

Btw I changed python to python3 several places in the documentation, as Python 2 will be removed in the next version of Kivy.

[matham]

Just FYI, for a PR, [build wheel XXX] must be in the PR title, not commit message. That's why it's not building wheels. The reason is that there's no easy way to get the commit message of a PR from the actions.

Also, from piwheels:

Why are the wheel files are tagged with Armv6 and Armv7? What about Armv8?

While the hardware in Raspberry Pi 1 and Zero is Armv6, Pi 2 is Armv7, Pi 3 and 4 are Armv8, the Raspbian operating system images provided on raspberrypi.org are made to support all three architectures (and are compatible with all Raspberry Pi products to date) by operating in Armv6 userland.

However, wheels built on a Raspberry Pi 2 or 3 are tagged armv7l. Since wheels built on a Raspberry Pi 3 will work on a Pi 4, 3, 2, 1 or Zero, we simply provide Pi 3-built wheels renamed armv6l, with a few exceptions (some packages, like opencv and tensorflow, are built with optimisations available on Pi 3).

https://www.piwheels.org/faq.html

I now wonder if we should also make named copies like that for v6? Have you tried these wheels on a pi3 e.g.?

[Lauszus]

Just FYI, for a PR, [build wheel XXX] must be in the PR title, not commit message. That's why it's not building wheels. The reason is that there's no easy way to get the commit message of a PR from the actions.

Yes it does trigger, but it builds at my fork instead: https://github.com/Lauszus/kivy/commit/14c1783d1cb9f70ee90a3b7ccf1892f76e6df030/checks?check_suite_id=369873129 :)

https://www.piwheels.org/faq.html
I now wonder if we should also make named copies like that for v6? Have you tried these wheels on a pi3 e.g.?

Yes, I'll update the script, so it creates a copy with the armv6l suffix. That is what piwheels seems to do:

$ md5sum Kivy-1.11.1-cp37-cp37m-linux_armv*
5a38955577867e1ccfa3451f9537bce9  Kivy-1.11.1-cp37-cp37m-linux_armv6l.whl
5a38955577867e1ccfa3451f9537bce9  Kivy-1.11.1-cp37-cp37m-linux_armv7l.whl

Note that I downloaded the wheels from: https://www.piwheels.org/simple/kivy/

[Lauszus]

Hmm Jessie fails, as the Cython version we currently use does not support Python 3.4: https://github.com/Lauszus/kivy/commit/ee83dfbfa602743d02bb3ceb4d23995037f44d86/checks?check_suite_id=369862780.

I've added Stretch, so that should cover most of the Raspberry Pi users.

[matham]

OK, one more thing, rather than using depends, you can specify a build matrix e.g. https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idstrategy. Then you can just list all the images in the matrix and reference it in the build step as in the example in the link.

[Lauszus]

@matham I have now modified the script to use a build matrix instead.

[Lauszus]

Btw we could also compile wheels specifically for the Raspberry Pi 1-3, as you can get the Broadcom drivers here: https://github.com/raspberrypi/firmware.

[matham]

Just these small changes and then we're good.

[matham]

Btw we could also compile wheels specifically for the Raspberry Pi 1-3, as you can get the Broadcom drivers here: https://github.com/raspberrypi/firmware.

If I read the piwheels FAQ correctly, shouldn't these wheels we generate automatically work on the other RPis already without needing to do anything more?

[Lauszus]

If I read the piwheels FAQ correctly, shouldn't these wheels we generate automatically work on the other RPis already without needing to do anything more?

Currently they do not: piwheels/packages#5.

[Lauszus]

Hmm would it be better to create a symlink for the armv6l wheels or are you happy with the copies?

[matham]

Thank you for all your work on this!

Hmm would it be better to create a symlink for the armv6l wheels or are you happy with the copies?

I'm not sure we can upload symlinks to the server, so copies are better for now.

[Lauszus]

You're welcome. Can't wait for the wheels to start popping up here: https://kivy.org/downloads/ci/raspberrypi/kivy/ :)

[Lauszus]

FYI I just tested the wheel on a Raspberry Pi 3 and it works fine when X is installed. I do not have physical access to it, so I can not check the performance. Anyway I'll look into cross compiling for Raspberry 1-3 using the Broadcom drivers next.

[matham]

Nice. Could one wheel support all the versions using the Broadcom drivers?

[Lauszus]

@matham no, as the Raspberry Pi 4 do not have them. FYI I have opened up a PR with for compiling wheels for Raspberry Pi 1-3 as well: #6662.