Add CI for a Python wheel release to PyPi #72

mballance · 2019-11-19T02:11:02Z

Signed-off-by: Matthew Ballance matt.ballance@gmail.com

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mballance · 2019-11-19T02:17:32Z

I've updated the Azure Pipelines specification to only authenticate and upload to PyPi when the build is done from the 'master' branch. It's pretty unlikely that we want PR builds to make uploads to PyPi.

In order to enable automated uploads to pypi.org, the following must be done in the Azure Pipelines Builds section. Before doing so, register for a free account on pypi.org

Project Settings -> Service Connections (it's under the Pipelines heading)
New Service connection -> Python package upload
- Connection name: pypi
- Python repository url for upload: https://pypi.python.org/pypi
- EndpointName: pyboolector
- Username: pypi.org username
- Password: pypi.org password
Select OK

The CI script can now authenticate a connection to PyPi.org via the information saved in the service connection configuration.

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner · 2019-12-03T20:12:44Z

@mballance Thanks a lot for the work and sorry for the delay! We'll have a look and setup pypi ASAP.

mpreiner · 2020-03-26T17:54:57Z

@mballance Sorry that this takes so long. I had a brief look (will dive into it more soon). However, I have one question: Why did you choose to use Apache instead of sticking with MIT that comes with Boolector? This may cause some confusion since only the code for packaging pyboolector for pypi would be under Apache, but Boolector and pyboolector are still MIT, which is currently not reflected in setup.py.

mballance · 2020-03-26T18:00:59Z

@mpreiner, looks like an oversight on my part. I'll be happy to correct the license to align with the MIT that Boolector uses.

-Matthew

mpreiner · 2020-03-26T18:02:16Z

Thanks @mballance!

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner

@mballance as discussed offline here is the first pass over the PR. It seems that the build script can be simplified quite a bit and I'd like to include CaDiCaL, which is the default SAT solver for Boolector and provides better performance. Thanks again for your work and sorry for the delay!

pypi/LICENSE

pypi/build.sh

…ge ; Update lingeling commit hash Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mballance · 2021-03-14T23:05:14Z

Hi @mpreiner,
I've updated the PR with your requested changes, and also added in a couple of changes from the separate CI build -- macOS release, and improved versioning.

The PR now uses the contrib/setup scripts for building dependencies. I updated the lingeling commit hash to pick up the fix for a name-collision bug (Avoid conflict between clone in sched.h and the Lingeling variables "clone" at global scope arminbiere/lingeling#2) that impacted the PyPi builds
Both CaDiCaL and Lingeling SAT solvers are provided
Simplified the build script for Linux, and made a few updates to Docker image and CI config

Thanks and Best Regards,
Matthew

aytey · 2021-03-14T23:14:38Z

My Lingeling PR isn’t merged; how are you using the commit hash from that?

Could we do something here like we do with the Windows patches to make this self-contained within the Boolector repo?

pypi/build.sh

mballance · 2021-03-14T23:17:29Z

My Lingeling PR isn’t merged; how are you using the commit hash from that?

Could we do something here like we do with the Windows patches to make this self-contained within the Boolector repo?

Apologies for the confusion, @andrewvaughanj. I wanted to point @mpreiner to a description of the issue resolved by moving to the latest lingeling code. It appears the issue has been resolved independent of your PR. The clone variable is now named 'cloned'.

src/api/python/pyboolector.pyx

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

aytey · 2021-03-14T23:19:05Z

My Lingeling PR isn’t merged; how are you using the commit hash from that?
Could we do something here like we do with the Windows patches to make this self-contained within the Boolector repo?

Apologies for the confusion, @andrewvaughanj. I wanted to point @mpreiner to a description of the issue resolved by moving to the latest lingeling code. It appears the issue has been resolved independent of your PR. The clone variable is now named 'cloned'.

Ah! All good :)

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner

Thanks a lot for the changes! A few more comments before we are good to go.

pypi/setup.py

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner · 2021-03-23T20:02:47Z

pypi/build.sh

+for py in cp35-cp35m cp36-cp36m cp37-cp37m cp38-cp38 cp39-cp39; do
+  echo "Python: ${py}"
+  python=/opt/python/${py}/bin/python
+  cd ${BUILD_DIR}/pyboolector
+  rm -rf src
+  cp -r ${BUILD_DIR}/boolector/src/api/python src
+  sed -i -e 's/override//g' \
+     -e 's/noexcept/_GLIBCXX_USE_NOEXCEPT/g' \
+     -e 's/\(BoolectorException (const.*\)/\1\n    virtual ~BoolectorException() _GLIBCXX_USE_NOEXCEPT {}/' \
+       src/pyboolector_abort.cpp
+  mkdir -p src/utils
+  cp ${BUILD_DIR}/boolector/src/*.h src
+  cp ${BUILD_DIR}/boolector/src/utils/*.h src/utils
+  $python ./src/mkenums.py ./src/btortypes.h ./src/pyboolector_enums.pxd
+  $python setup.py sdist bdist_wheel
+done


Just out of curiosity: Why can't the library produced by ./configure --python be used here?

The only way I could find to produce a 'wheel' (Python distribution package) that PyPi would accept was to use the Python build commands via setup.py. I think part of this is due to the package auditing that exists to ensure that packages posted on PyPi will truly be compliant with the platform specifications they purport to be compatible with.

BTW, one option going forward (thinking Bitwuzla) would be to build all Python packages via a setup.py. The CMakeLists.txt would be slightly different, and would invoke Python on the setup.py instead of directly building the shared library. The advantage would be a single way of building the Python package.

Oh ok. Do you have experience with scikit-build? Looks like this is allows better integration with a cmake build system.

Unfortunately, I don't. I think I looked at scikit-build early in my process of learning about building Python extensions, and think my conclusion was that it was optimized for certain types of extension but lacked flexibility. Certainly not in a position to offer a knowledgeable opinion, though.

I've been working on building Python extensions as part of the regular build in the context of a different project. Thus far, it doesn't look too difficult. Here's the CMakeList fragment that invokes the actual build:
https://github.com/PSSTools/pssparser/blob/67fdb8d0cd41874f80f6e8c1acb60a83ac6cc156/ext/CMakeLists.txt#L7-L20

This, in turn, is driven by a script run inside the 'manylinux2010' Docker image to produce all the Python-version variants:
https://github.com/PSSTools/pssparser/blob/67fdb8d0cd41874f80f6e8c1acb60a83ac6cc156/.github/workflows/pypi.sh#L26-L38

Fortunately, Makefile dependencies cause only the Python extension (and not all the other code) to be rebuilt for each Python version

aytey · 2021-03-23T20:17:47Z

pypi/build.sh

+  sed -i -e 's/override//g' \
+     -e 's/noexcept/_GLIBCXX_USE_NOEXCEPT/g' \
+     -e 's/\(BoolectorException (const.*\)/\1\n    virtual ~BoolectorException() _GLIBCXX_USE_NOEXCEPT {}/' \
+       src/pyboolector_abort.cpp


What's this bit for?

In order to support a broad range of Linux distributions, pre-compiled Python libraries are built on specific older Linux distros with older compilers (Python defines a manylinux docker image based on CentOS). The supported compiler versions have limits on the C++ dialects supported (override, noexcept as a keyword, etc). This filter patches the code to comply with the compiler-supported C++.

mpreiner · 2021-03-26T17:06:10Z

pypi/azure-pipelines-pypi.yml

@@ -0,0 +1,113 @@
+
+jobs:


Is there a way to only run the upload to pypi if we do a new release? I think PyPi has upload limits and we should not upload a new version on every commit.

There should be, as long as we can detect creation of a tag or something else that marks the creation of a release. I always find I have to do a bit of Googling and playing around to get this to work, but there typically is a way. How are releases marked for Boolector? A tag? A branch?
That said, I've not detected any upload limits with PyPi on any of the projects I'm using it with.

Releases are tagged, but there is also the releases page: https://github.com/Boolector/boolector/releases

If Azure doesn't have this kind of functionality we could also think of switching the PyPi upload to GitHub actions, which supports running workflows only for specific actions such as creating a release.

About the size limit: I was also not aware of that, but it was mentioned here: cvc5/cvc5#5657 (comment)

mpreiner

@mballance is there anything else to do other than what you wrote here once the PR is merged?

mballance · 2021-04-01T23:52:56Z

@mballance is there anything else to do other than what you wrote here once the PR is merged?

@mpreiner, other than configuring PyPi upload (referenced link), we should be good to go. Please let me know the PyPi username that will be doing uploads, since I think I'll need to add it to the PyPi project.

Add implementation files for a Python wheel release to PyPi

c6d3f4c

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

temporarily comment out authentication to get past PR build

8176a54

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mballance mentioned this pull request Nov 20, 2019

Release boolector as a PyPi package #49

Closed

Align license terms with Boolector (MIT)

1be9a76

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner suggested changes Mar 12, 2021

View reviewed changes

mballance added 4 commits March 14, 2021 11:27

Clean up and simplify build script ; Move to manylinux2010 docker ima…

9557676

…ge ; Update lingeling commit hash Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

Merge remote-tracking branch 'upstream/master' into pyboolector

002301d

More tweaks to build process

a022318

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

Move to ubuntu-latest image

b09d1ef

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

aytey reviewed Mar 14, 2021

View reviewed changes

pypi/build.sh Outdated Show resolved Hide resolved

aytey reviewed Mar 14, 2021

View reviewed changes

src/api/python/pyboolector.pyx Outdated Show resolved Hide resolved

Remove redundant exit-code checks

9e1fb33

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mballance added 2 commits March 14, 2021 16:19

Remove language-level specification

427c0a6

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

Move language-level settings to setup.py

0a9afaf

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

mpreiner reviewed Mar 15, 2021

View reviewed changes

pypi/setup.py Show resolved Hide resolved

pypi/setup.py Outdated Show resolved Hide resolved

pypi/setup.py Outdated Show resolved Hide resolved

mballance added 2 commits March 15, 2021 18:32

Enhance libboolector check ; List Python 3.9 as supported ; cleanup

47dd924

Signed-off-by: Matthew Ballance <matt.ballance@gmail.com>

Merge branch 'master' into pyboolector

b20b68d

mpreiner reviewed Mar 23, 2021

View reviewed changes

aytey reviewed Mar 23, 2021

View reviewed changes

mpreiner reviewed Mar 26, 2021

View reviewed changes

mpreiner approved these changes Apr 1, 2021

View reviewed changes

Merge branch 'master' into pyboolector

515d067

mpreiner merged commit 4a46a95 into Boolector:master Apr 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI for a Python wheel release to PyPi #72

Add CI for a Python wheel release to PyPi #72

mballance commented Nov 19, 2019

mballance commented Nov 19, 2019

mpreiner commented Dec 3, 2019

mpreiner commented Mar 26, 2020

mballance commented Mar 26, 2020

mpreiner commented Mar 26, 2020

mpreiner left a comment

mballance commented Mar 14, 2021

aytey commented Mar 14, 2021

mballance commented Mar 14, 2021

aytey commented Mar 14, 2021

mpreiner left a comment

mpreiner Mar 23, 2021

mballance Mar 24, 2021

mballance Mar 24, 2021

mpreiner Mar 26, 2021 •

edited

Loading

mballance Mar 26, 2021

aytey Mar 23, 2021

mballance Mar 24, 2021

mpreiner Mar 26, 2021

mballance Mar 26, 2021

mpreiner Mar 26, 2021 •

edited

Loading

mpreiner left a comment

mballance commented Apr 1, 2021

Add CI for a Python wheel release to PyPi #72

Add CI for a Python wheel release to PyPi #72

Conversation

mballance commented Nov 19, 2019

mballance commented Nov 19, 2019

mpreiner commented Dec 3, 2019

mpreiner commented Mar 26, 2020

mballance commented Mar 26, 2020

mpreiner commented Mar 26, 2020

mpreiner left a comment

Choose a reason for hiding this comment

mballance commented Mar 14, 2021

aytey commented Mar 14, 2021

mballance commented Mar 14, 2021

aytey commented Mar 14, 2021

mpreiner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpreiner Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpreiner Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

mpreiner left a comment

Choose a reason for hiding this comment

mballance commented Apr 1, 2021

mpreiner Mar 26, 2021 •

edited

Loading

mpreiner Mar 26, 2021 •

edited

Loading