Wheels for all

[altendky]

There's #3747 for just Windows wheels but really they should be available for all. Here's a lazy shortcut way that I just did as a demo for someone.

venv/bin/romp --command 'git clone https://github.com/sqlalchemy/sqlalchemy; python -m pip install cibuildwheel; python -m cibuildwheel --output-dir wheelhouse sqlalchemy' --version 3.7 --artifact-paths wheelhouse

https://dev.azure.com/altendky/romp-on/_build/results?buildId=5224

I don't so much encourage use of https://github.com/altendky/romp for a 'real' implementation but that's what I was playing with at the time and it makes a quick external-to-the-project example of a thing that works.

[altendky]

Direct artifact link while the build lasts: https://dev.azure.com/altendky/f1722f91-62fe-4a15-8937-252c96b31292/_apis/build/builds/5224/artifacts?artifactName=artifacts&api-version=5.2-preview.5&%24format=zip

[zzzeek]

I dont have personally have the resources to build wheels for SQLAlchemy or all the setup required to run "cibuildwheel" ('currently it supports Azure Pipelines, Travis CI, AppVeyor, and CircleCI -" i use none ot these) SQLAlchemy has thousands of users and I would hope that someone out there has the motivation and interest to handle this, especially the windows stuff.

Basically, I release the source on pypi, then (volunteer) who has access would maintain jobs on those 3rd party servers to build the wheels. when things break as they will on a regular basis, that person can handle getting it working again.

[altendky]

@zzzeek I hope I didn't come off the wrong way. I understand that a 'real' solution takes more than one command line. Just figured that since there wasn't an existing ticket for wheels for all platforms I'd go ahead and share this while entering the ticket. I can't say that I personally use SQLAlchemy but I know that lots of other people do and we certainly appreciate everything that you've given us.

[zzzeek]

no worries, it's frustrating because yes I know there are travis -etc services that provide the hardware to do the builds and in #3747 it was incredibly close to working, however I couldn't get it working the way I wanted, I was drowning in weird powershell stuff I didn't understand, and it was sucking down all my time, and I've kind of made a boundary that this is something that other people can definitely handle, if I ever had dedicated volunteers; like 1% as many volunteers as like cPython has for example.

Really, the SQLAlchemy release process needs to be a lot more heavyweight since ideally you wouldn't release until the wheels all build and all pass CI themselves, which is a huge ordeal to set up. That was part of the controversy; that putting source on pypi alone and then adding the wheels later is technically "wrong" even though in the vast majority of cases that approach is fine.

[altendky]

CI is definitely a time suck. I also agree that the proper thing is to test the wheels. I've got a Twisted PR I need to get back to for just that. Not that it makes it not take effort, but with Azure being a one stop CI service at least there's only one system to deal with. If anyone wants to take this on I would usually at least have time for guidance or such. Or maybe I free up a bit and do something myself. Heh...

I didn't see CI config beyond the tox.ini. Is there a quick link to that or would I just need to learn Gerrit? Or is it Jenkins?

[zzzeek]

I have a jenkins/gerrit combination running:

https://gerrit.sqlalchemy.org/
https://jenkins.sqlalchemy.org/

you may notice the jenkins has three build followers which is in fact some computing resources, but that's because i have 15 different databases running on them in containers which is a heavy lift for one of the third party CI services. for builds, it would be nice if these ran on someone else's hardware.

[CaselIT]

#5241 will close this.