Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) #869

webknjaz · 2019-02-11T18:27:58Z

Closes #793

In its current state the changed added in this PR enable Travis CI build and publish OS-specific wheel distributions which are compliant with a manylinux1 standard.

This PR has a few intentionally incomplete bits:

it requires a PyPI password to be filled in
it requires uncommenting deploy stage clause so that publishing would only happen for tagged commits

Considerations

For simplicity, this PR reuses image by pyca/cryptography which inherits the official image by pypa/manylinux1 and adds a pre-compiled openssl 1.1.1 on top of it so we don't have to.

As a further build time performance improvement I see how we can improve things by publishing our own image to a public registry (Docker Hub or better quay.io) but it's out of scope of this PR.

Credits

This patch is based on the prior art by @beniwohli and my own experience of building manylinux1 wheels for other projects.

techtonik · 2019-02-11T20:24:06Z

Have you checked the status of manylinux2010 - it seems to be the way to go pypa/manylinux#179

webknjaz · 2019-02-11T21:02:29Z

Yeah, I know. pip>19 now supports it. And the latest auditwheel has it in the list of platforms.
But first I want to get manylinux1 done. It's more important since pip on user machines is often outdated and is unable to match manulinux2010 which would lead to falling back to sdist if manylinux1 is not shipped.

webknjaz · 2019-02-12T01:06:15Z

.travis.yml

+stages:
+- test
+- name: *deploy_stage
+  if: 1=1 OR tag IS present OR type IN (cron, api)


@jdavid
We'll need to apply

Suggested change

if: 1=1 OR tag IS present OR type IN (cron, api)

if: tag IS present OR type IN (cron, api)

once password is updated.

webknjaz · 2019-02-12T01:08:43Z

.travis.yml

+      password:
+        # Encrypted with `travis encrypt -r aio-libs/aiohttp --api-endpoint 'https://api.travis-ci.com/'`:
+        secure: >-
+          TODO: REPLACE THIS WITH AN ENCRYPTED PASSWORD


@jdavid please generate the encrypted string using travis encrypt -r libgit2/pygit2 command.

And then you can use this page to update in via UI right in this PR: https://github.com/webknjaz/pygit2/edit/feature/manylinux1-binary-wheels/.travis.yml?pr=%2Flibgit2%2Fpygit2%2Fpull%2F869

webknjaz · 2019-02-12T01:10:45Z

@jdavid this is almost ready to be merged.
You need to update encrypted PYPI password and we'll need to update deploy stage appearance condition (which I didn't do for demo purposes).

webknjaz · 2019-02-12T08:50:30Z

(PR close/reopen trick is just to trigger CI restart)

As recommended @ https://libgit2.org/docs/guides/build-and-link/

Add test target. 0.28.0 (2019-03-19) ------------------------- - Upgrade to libgit2 0.28 `#878 <https://github.com/libgit2/pygit2/issues/878>`_ - Add binary wheels for Linux `#793 <https://github.com/libgit2/pygit2/issues/793>`_ `#869 <https://github.com/libgit2/pygit2/pull/869>`_ `#874 <https://github.com/libgit2/pygit2/pull/874>`_ `#875 <https://github.com/libgit2/pygit2/pull/875>`_ `#883 <https://github.com/libgit2/pygit2/pull/883>`_ - New ``pygit2.Mailmap``, see documentation `#804 <https://github.com/libgit2/pygit2/pull/804>`_ - New ``Repository.apply(...)`` wraps ``git_apply(..)`` `#841 <https://github.com/libgit2/pygit2/issues/841>`_ `#843 <https://github.com/libgit2/pygit2/pull/843>`_ - Now ``Repository.merge_analysis(...)`` accepts an optional reference parameter `#888 <https://github.com/libgit2/pygit2/pull/888>`_ `#891 <https://github.com/libgit2/pygit2/pull/891>`_ - Now ``Repository.add_worktree(...)`` accepts an optional reference parameter `#814 <https://github.com/libgit2/pygit2/issues/814>`_ `#889 <https://github.com/libgit2/pygit2/pull/889>`_ - Now it's possible to set SSL certificate locations `#876 <https://github.com/libgit2/pygit2/issues/876>`_ `#879 <https://github.com/libgit2/pygit2/pull/879>`_ `#884 <https://github.com/libgit2/pygit2/pull/884>`_ `#886 <https://github.com/libgit2/pygit2/pull/886>`_ - Test and documentation improvements `#873 <https://github.com/libgit2/pygit2/pull/873>`_ `#887 <https://github.com/libgit2/pygit2/pull/887>`_ Breaking changes: - Now ``worktree.path`` returns the path to the worktree directory, not to the `.git` file within `#803 <https://github.com/libgit2/pygit2/issues/803>`_ - Remove undocumented ``worktree.git_path`` `#803 <https://github.com/libgit2/pygit2/issues/803>`_ 0.27.4 (2019-01-19) ------------------------- - New ``pygit2.LIBGIT2_VER`` tuple `#845 <https://github.com/libgit2/pygit2/issues/845>`_ `#848 <https://github.com/libgit2/pygit2/pull/848>`_ - New objects now support (in)equality comparison and hash `#852 <https://github.com/libgit2/pygit2/issues/852>`_ `#853 <https://github.com/libgit2/pygit2/pull/853>`_ - New references now support (in)equality comparison `#860 <https://github.com/libgit2/pygit2/issues/860>`_ `#862 <https://github.com/libgit2/pygit2/pull/862>`_ - New ``paths`` optional argument in ``Repository.checkout()`` `#858 <https://github.com/libgit2/pygit2/issues/858>`_ `#859 <https://github.com/libgit2/pygit2/pull/859>`_ - Fix speed and windows package regression `#849 <https://github.com/libgit2/pygit2/issues/849>`_ `#857 <https://github.com/libgit2/pygit2/issues/857>`_ `#851 <https://github.com/libgit2/pygit2/pull/851>`_ - Fix deprecation warning `#850 <https://github.com/libgit2/pygit2/pull/850>`_ - Documentation fixes `#855 <https://github.com/libgit2/pygit2/pull/855>`_ - Add Python classifiers to setup.py `#861 <https://github.com/libgit2/pygit2/pull/861>`_ - Speeding up tests in Travis `#854 <https://github.com/libgit2/pygit2/pull/854>`_ Breaking changes: - Remove deprecated `Reference.get_object()`, use `Reference.peel()` instead 0.27.3 (2018-12-15) ------------------------- - Move to pytest, drop support for Python 3.3 and cffi 0.x `#824 <https://github.com/libgit2/pygit2/issues/824>`_ `#826 <https://github.com/libgit2/pygit2/pull/826>`_ `#833 <https://github.com/libgit2/pygit2/pull/833>`_ `#834 <https://github.com/libgit2/pygit2/pull/834>`_ - New support comparing signatures for (in)equality - New ``Submodule.head_id`` `#817 <https://github.com/libgit2/pygit2/pull/817>`_ - New ``Remote.prune(...)`` `#825 <https://github.com/libgit2/pygit2/pull/825>`_ - New ``pygit2.reference_is_valid_name(...)`` `#827 <https://github.com/libgit2/pygit2/pull/827>`_ - New ``AlreadyExistsError`` and ``InvalidSpecError`` `#828 <https://github.com/libgit2/pygit2/issues/828>`_ `#829 <https://github.com/libgit2/pygit2/pull/829>`_ - New ``Reference.raw_name``, ``Reference.raw_shorthand``, ``Tag.raw_name``, ``Tag.raw_message`` and ``DiffFile.raw_path`` `#840 <https://github.com/libgit2/pygit2/pull/840>`_ - Fix decode error in commit messages and signatures `#839 <https://github.com/libgit2/pygit2/issues/839>`_ - Fix, raise error in ``Repository.descendant_of(...)`` if commit doesn't exist `#822 <https://github.com/libgit2/pygit2/issues/822>`_ `#842 <https://github.com/libgit2/pygit2/pull/842>`_ - Documentation fixes `#821 <https://github.com/libgit2/pygit2/pull/821>`_ Breaking changes: - Remove undocumented ``Tag._message``, replaced by ``Tag.raw_message``

webknjaz mentioned this pull request Feb 11, 2019

Ship wheels for Linux #793

Closed

webknjaz force-pushed the feature/manylinux1-binary-wheels branch from f0d23e8 to c2ac65a Compare February 11, 2019 19:52

webknjaz closed this Feb 11, 2019

webknjaz reopened this Feb 11, 2019

webknjaz force-pushed the feature/manylinux1-binary-wheels branch from c2ac65a to 942ef20 Compare February 11, 2019 19:58

webknjaz mentioned this pull request Feb 11, 2019

Depend on pygit2 that corresponds to the installed libgit2 #847

Closed

beniwohli and others added 18 commits February 12, 2019 01:59

let's give this a try

b470b1a

vars

89d62fd

Rearrange travis config

5023308

Bump libgit2 source to v0.27

a0c2679

Fix building manylinux1 wheels

5d184dd

Don't execute install step in docker jobs

94edb05

Fix referring pip bin var

fbcef81

Fix env vars in travis

eaedbf8

Upgrade PyPy to v6.0

976d544

Run pip in parallel

948872c

Adjust tests to succeed under PyPy v6.0

7a5bc52

Put manylinux1 jobs into a separate stage

8db38f7

Add deploy step stub to travis

5f2c250

Undo async in wheel build script

ae1ee1c

Create wheelhouse dir on build start

dc801b7

Be more precise during wheel processing

548ecf4

Add a wrapper script for building several archs

518984d

Add a deploy step stub to Travis

029165c

webknjaz force-pushed the feature/manylinux1-binary-wheels branch from 30947c8 to 029165c Compare February 12, 2019 00:59

webknjaz added 2 commits February 12, 2019 02:01

Save final wheels to dist dir directly

d9d40d2

Add deploy stage conditional stub

e184187

webknjaz commented Feb 12, 2019

View reviewed changes

webknjaz added 3 commits February 12, 2019 02:42

List the whole wheelhouse folder

ee78fde

List full dist folder in travis

6b55b79

Copy travis config owner to dist dir

b530d20

webknjaz mentioned this pull request Feb 12, 2019

TypeError: don't know how to make test from: <CompiledFFI object at 0x7f43c00693e8> #868

Closed

webknjaz closed this Feb 12, 2019

webknjaz reopened this Feb 12, 2019

webknjaz mentioned this pull request Feb 13, 2019

[TODO] Provide an out-of-the-box integration with libgit2 sanitizers/octomachinery#4

Open

webknjaz changed the title ~~Build manylinux1 OS-specific binary wheels~~ Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) Feb 13, 2019

webknjaz added 5 commits February 13, 2019 17:33

Compile libgit2 in threadsafe mode

18abf68

As recommended @ https://libgit2.org/docs/guides/build-and-link/

Compile libgit2 in "Release" build type mode

e1ebe52

Detect custom OpenSSL properly

457b1d1

Extend build flags

57da1d5

Build libgit2 with libssh2 v1.8.0

4a50c76

jdavid merged commit 4a50c76 into libgit2:master Feb 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) #869

Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) #869

webknjaz commented Feb 11, 2019 •

edited

Loading

techtonik commented Feb 11, 2019

webknjaz commented Feb 11, 2019 •

edited

Loading

webknjaz Feb 12, 2019

webknjaz Feb 12, 2019

webknjaz commented Feb 12, 2019

webknjaz commented Feb 12, 2019

	if: 1=1 OR tag IS present OR type IN (cron, api)
	if: tag IS present OR type IN (cron, api)

Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) #869

Build manylinux1 OS-specific binary wheels in Travis (CI) and publish them to PyPI (CD) #869

Conversation

webknjaz commented Feb 11, 2019 • edited Loading

Considerations

Credits

techtonik commented Feb 11, 2019

webknjaz commented Feb 11, 2019 • edited Loading

webknjaz Feb 12, 2019

Choose a reason for hiding this comment

webknjaz Feb 12, 2019

Choose a reason for hiding this comment

webknjaz commented Feb 12, 2019

webknjaz commented Feb 12, 2019

webknjaz commented Feb 11, 2019 •

edited

Loading

webknjaz commented Feb 11, 2019 •

edited

Loading