TravisCI automation for building Python binaries

[dtreskunov]

Finally my branch is in a good state. This should be a good addition allowing frequent releases for many platforms with minimal manual work.

Note on Linux builds

Python whl packages are cross-compiled for Linux platforms supported
by dockcross. It would be trivial to add more platforms.

• linux-armv6: should work on RPi Zero
• linux-armv7: tested on RPi 3
• manylinux2010-x64: tested on Debian buster

Note that dockcross/windows-shared-x64 doesn't support building OpenFST
due to certain syscalls not being supported in the MinGW environment.
See Note below about Windows builds.

QEMU is registered as the binfmt handler inside Travis CI Docker host
(dockcross images already come with the qemu binaries in /usr/bin). This
allows the Docker container to run foreign binaries (e.g. x86-64 host becomes
able to run ARM binaries), and is needed for some Autoconf scripts to
work correctly.

Intermediate build stages are pushed to Docker Hub to work around the 50 min
limit imposed by Travis CI on unpaid users. Initial build will likely time
out, but can be restarted in Travis UI and will use the stages in the Dockerfile
which were built by previous jobs as cache. Fully-cached builds take 5-10
minutes depending on the number of Python versions we want to build for.

docker build -t vosk-api-build:linux-armv7 --build-arg DOCKCROSS_IMAGE=linux-armv7 --build-arg OPENBLAS_TARGET=ARMV7 .

docker build -t vosk-api-build:manylinux2010-x64 --build-arg DOCKCROSS_IMAGE=manylinux2010-x64 --build-arg OPENBLAS_TARGET=NEHALEM .

docker run --rm -v $(pwd):/io vosk-api-build:linux-armv7 /io/travis/build-wheels.sh

docker run --rm -v $(pwd):/io vosk-api-build:manylinux2010-x64 /io/travis/build-wheels.sh

Note on Windows builds

Windows releases are compiled using Visual Studio 15 2017 following the
official guide from the Kaldi project.

Windows builds use Windows VMs provided by Travis CI. Surprisingly,
commands specified in .travis-ci.yml are executed in a Git Bash (MinGW)
shell, so the scripts are implemented in Bash. Like in Linux builds,
the build is broken up into several steps and will cache intermediate
artifacts. So if a Travis CI job fails due to the 50-minute timeout,
restarting the job will pick up artifacts produced by earlier steps
(if they succeeded).

Note on deploying of build artifacts

Uploads whl files to PyPI if the build is for a tagged commit. This is
conditional on TWINE_USERNAME and TWINE_PASSWORD environment variables
being set.

Also creates a draft release on GitHub Releases if GITHUB_TOKEN is set.

The following variables affect the build, and should be defined in your
Travis project settings:

• DOCKER_REGISTRY: name of your registry on Docker Hub, e.g. alphacep
• DOCKER_USERNAME: your Docker Hub username
• DOCKER_PASSWORD: your Docker Hub password
• GITHUB_TOKEN: (optional) GitHub token authorizing Travis to create Releases in this project
• TWINE_USERNAME: (optional) your PyPI username
• TWINE_PASSWORD: (optional) your PyPI password
• BUILD_OPENFST: (optional, Windows-only) if non-empty, OpenFST will be built *
• BUILD_KALDI: (optional, Windows-only) if non-empty, Kaldi will be built *
• REBUILD_OPENFST: (optional, Windows-only) if non-empty, OpenFST will be rebuilt from scratch *
• REBUILD_KALDI: (optional, Windows-only) if non-empty, Kaldi will be rebuilt from scratch *

(*) even if cache contains artifacts from a successful build

[nshmyrev]

Sounds great, thank you! I'll try to deploy it coming days.

[dtreskunov]

Also don't ignore my other PR with methods for getting model frequency and controlling resampling.

…

On Mon, Feb 17, 2020, 10:53 Nickolay V. Shmyrev ***@***.***> wrote: Sounds great, thank you! I'll try to deploy it coming days. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=AA2RDSHZYQAC35MO5Y4NMSLRDLMKXA5CNFSM4KWL7LLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL7L5FY#issuecomment-587120279>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2RDSDX2OKBPQ3KIMJAZ5TRDLMKXANCNFSM4KWL7LLA> .

[nshmyrev]

Hi

Ok, I reviewed, thanks a lot for such a titanic work on making things working. I prefer we do this way:

1. linux build - use manylinux-based docker with prebuilt kaldi, only build python in travis, like we are doing now
2. armv6/v7 build - use dockcross-based docker with prebuilt kaldi just like above
3. win64 - use prebuilt dlls hosted on our server and only build a python module like above
4. No need to github/pypi provision yet, I want to do it slightly different with secret keys instead of environment variables, it could be a separate patch

Please let me know what do you think on this.