Pythemis: introduce pyproject.toml
#1006
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since setup.py is deprecated, let's try moving to the pyproject.toml and configuring it with the same data as in setup.py. Use setuptools as a backend for no particular reasons ¯\_(ツ)_/¯, just because the name is familiar and we have no reasons to not use it or use something else. Keep the old setup.py for backward compatibility so old systems can try to build the package. For now, keep 0.14.0, we will bump the version in another PR.
According to this [1] article, the correct command is
pip install .
in the project's root. Let's try that. Also, the other option is
python -m build --wheel
which builds the package but doesn't install it. We can provide
something like `pythemis_build` for it for example.
[1]: https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/
With many hours and docker containers I tested that themis actually works up to python 3.4. The other versions require some changes in makefile so they are more like "grey area". However, python3.5 is deprecated and it produces warning like "DEPRECATION: Python 3.5 reached the end of its life on..." so many libraries don't support it. Instead they start with 3.6 which will do as well, I guess. Though, actually python3.6 is also deprecated [1]. The same will be true for python3.7 in a couple of days (Jun 27 2023), so the question is, should we declare support of these versions? [1]: https://devguide.python.org/versions/
G1gg1L3s
requested review from
vixentael,
ilammy,
shadinua and
Lagovas
as code owners
Lagovas
reviewed
Jun 15, 2023
There was a problem hiding this comment.
So the question is should we declare support of these versions? Or should we declare support of the whole range python 3.4-3.11?
on the one side we test only subset of supported versions, on the other side limiting versions can hide pythemis in PYPI in search results when it filtered by version even if it works. And I'm not sure, but guess that I met situation when I tried to install package via pip to my virtualenv with python version X, but package didn't specify that version. And pip logged that cannot find package with required version support.
So, my vote for specifying all working versions to not limit users from installing it via pip
| @@ -598,7 +598,7 @@ ifdef PIP_VERSION | |||
| PIP_THEMIS_INSTALL := $(shell pip freeze |grep themis) | |||
| endif | |||
|
|
|||
| pythemis_install: CMD = cd src/wrappers/themis/python/ && python3 setup.py install --record files3.txt | |||
| pythemis_install: CMD = cd src/wrappers/themis/python/ && pip3 install . | |||
There was a problem hiding this comment.
with which version of pip3 it works? if it doesn't work with pip version from our supported distros (centos 7/8, last 3 versions of ubuntu/debian), so we should add here pip3 install -U pip to update system version pip to newer
There was a problem hiding this comment.
Hmm, it requires additional testing, preferably not manual as I did. But I guess for now we can add this command just to be sure.
There was a problem hiding this comment.
I'm thinking, maybe we should add that command to the prepare_tests_all? Updating pip each time pythemis_install is called doesn't sound good to me. For example, maybe there are reasons why users don't want to update pip.
As I understand, only we use prepare_tests_all in our testing environments, so putting this command there should prevent problems with pip on our test machines. At the same time, if users, who install pythemis from the sources, will have to fix their environments in controlled way, if something goes wrong.
There was a problem hiding this comment.
agree, better if we will add it in our own test environment. but starting installation via pip we should be sure that this approach supported by all supported OS, because approach with setup.py works for all of them
There was a problem hiding this comment.
Hmmm, another thought came to mind. The new pip versions (for example, in debian bookworm) don't want to install packages globally, so we will have to fix something (use virtual environments, pipx, etc). So, maybe it is better to postpone the change and fix only if it breaks?
Currently, it seems like a small change but it requires extensive testing on all our platforms.
Lagovas
approved these changes
Jun 18, 2023
G1gg1L3s
added a commit
that referenced
this pull request
Jun 21, 2023
* Fix rust issues (pin log, run bindgen) (#1005) * rust: Pin log version to =0.4.18 The 0.4.19 requires rustc 1.60, but currently we support 1.58. Pinning it is not a big deal since it's development dependecy for tests and examples. * rust: Regenerate and update lib.rs bindgen was updated again and changed something which resulted in new output (seems like some internal constants are removed). * Pythemis: introduce `pyproject.toml` (#1006) * pythemis: Add pyproject.toml Since setup.py is deprecated, let's try moving to the pyproject.toml and configuring it with the same data as in setup.py. Use setuptools as a backend for no particular reasons ¯\_(ツ)_/¯, just because the name is familiar and we have no reasons to not use it or use something else. Keep the old setup.py for backward compatibility so old systems can try to build the package. For now, keep 0.14.0, we will bump the version in another PR. * makefile: Use pyproject.toml for installing pythemis According to this [1] article, the correct command is pip install . in the project's root. Let's try that. Also, the other option is python -m build --wheel which builds the package but doesn't install it. We can provide something like `pythemis_build` for it for example. [1]: https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/ * pythemis: Update classifiers to Python3.6+ With many hours and docker containers I tested that themis actually works up to python 3.4. The other versions require some changes in makefile so they are more like "grey area". However, python3.5 is deprecated and it produces warning like "DEPRECATION: Python 3.5 reached the end of its life on..." so many libraries don't support it. Instead they start with 3.6 which will do as well, I guess. Though, actually python3.6 is also deprecated [1]. The same will be true for python3.7 in a couple of days (Jun 27 2023), so the question is, should we declare support of these versions? [1]: https://devguide.python.org/versions/ * pythemis: Extend range of supported py versions * Update changelog * Run and pin bindgen (#1008) * rust-themis: Update bindgen It updated and broke something again 🤦 * rust-themis: Pin bindgen version It is pretty unstable with its frequent releases, so let's pin it. * Update changelog * Bump wrapper versions to 0.15.0 (#1007) * changelog: Add 0.15.0 summary * themis-core: Update version * pythemis: Update version * pythemis: Fix 8-year old typo in AUTHORS :) * rbthemis: Update version * jsthemis: Update versions * wasm-themis: Update versions * android-themis: Update version * rust-themis: Update versions * react-native-themis: Update versions * pythemis: https in AUTHORS Co-authored-by: vixentael <vixentael@users.noreply.github.com> * rust-themis: Update bench versions Somehow missed that. * changelog: Forgot to mention rust 1.58 * changelog: Mention the new iteration count --------- Co-authored-by: vixentael <vixentael@users.noreply.github.com> * Bump embedded BoringSSL (#1004) * Bump BoringSSL and fix makefile This is not the latest BoringSSL version yet, because there are a couple of fixes. So, treat it as the first. Here we also fix our makefile because the BoringSSL team fixed bug with the strange behaviour of absolute path to symbols.txt [1]. [1]: https://boringssl.googlesource.com/boringssl/+/8c75ed046f799f1d8b805036b1dea9c5ec0a0fb5%5E%21/#F0 * Bump BoringSSL and fix opaque EVP As OpenSSL, BoringSSL made many types opaque, so it will require updating some of the code to not use fields. * Bump BoringSSL again and fix RSA The same issue - RSA type became opaque, so we need to use accessors similar to what Openssl had. * Bump BoringSSL once more This is (hoperfully) the last bump. This time without issues but we will see what CI says. * Make bignum_to_bytes accept const bignum* It will prevent some of the warnings. This function doesn't mutate bignum anyway. * Update changelog * boringssl: Bump once again * msys2: Update hashes temporarily This are test values because we will move the tag. But for now, let's just test it. * phpthemis: Update version for the sake of testing They will fail probably, but just out of curiosity let's try to run the tests. * Update date of the release Solstice! --------- Co-authored-by: vixentael <vixentael@users.noreply.github.com>
Lagovas
pushed a commit
that referenced
this pull request
Dec 5, 2023
* Pre-release Themis 0.15.0 (#1011) * Fix rust issues (pin log, run bindgen) (#1005) * rust: Pin log version to =0.4.18 The 0.4.19 requires rustc 1.60, but currently we support 1.58. Pinning it is not a big deal since it's development dependecy for tests and examples. * rust: Regenerate and update lib.rs bindgen was updated again and changed something which resulted in new output (seems like some internal constants are removed). * Pythemis: introduce `pyproject.toml` (#1006) * pythemis: Add pyproject.toml Since setup.py is deprecated, let's try moving to the pyproject.toml and configuring it with the same data as in setup.py. Use setuptools as a backend for no particular reasons ¯\_(ツ)_/¯, just because the name is familiar and we have no reasons to not use it or use something else. Keep the old setup.py for backward compatibility so old systems can try to build the package. For now, keep 0.14.0, we will bump the version in another PR. * makefile: Use pyproject.toml for installing pythemis According to this [1] article, the correct command is pip install . in the project's root. Let's try that. Also, the other option is python -m build --wheel which builds the package but doesn't install it. We can provide something like `pythemis_build` for it for example. [1]: https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/ * pythemis: Update classifiers to Python3.6+ With many hours and docker containers I tested that themis actually works up to python 3.4. The other versions require some changes in makefile so they are more like "grey area". However, python3.5 is deprecated and it produces warning like "DEPRECATION: Python 3.5 reached the end of its life on..." so many libraries don't support it. Instead they start with 3.6 which will do as well, I guess. Though, actually python3.6 is also deprecated [1]. The same will be true for python3.7 in a couple of days (Jun 27 2023), so the question is, should we declare support of these versions? [1]: https://devguide.python.org/versions/ * pythemis: Extend range of supported py versions * Update changelog * Run and pin bindgen (#1008) * rust-themis: Update bindgen It updated and broke something again 🤦 * rust-themis: Pin bindgen version It is pretty unstable with its frequent releases, so let's pin it. * Update changelog * Bump wrapper versions to 0.15.0 (#1007) * changelog: Add 0.15.0 summary * themis-core: Update version * pythemis: Update version * pythemis: Fix 8-year old typo in AUTHORS :) * rbthemis: Update version * jsthemis: Update versions * wasm-themis: Update versions * android-themis: Update version * rust-themis: Update versions * react-native-themis: Update versions * pythemis: https in AUTHORS Co-authored-by: vixentael <vixentael@users.noreply.github.com> * rust-themis: Update bench versions Somehow missed that. * changelog: Forgot to mention rust 1.58 * changelog: Mention the new iteration count --------- Co-authored-by: vixentael <vixentael@users.noreply.github.com> * Bump embedded BoringSSL (#1004) * Bump BoringSSL and fix makefile This is not the latest BoringSSL version yet, because there are a couple of fixes. So, treat it as the first. Here we also fix our makefile because the BoringSSL team fixed bug with the strange behaviour of absolute path to symbols.txt [1]. [1]: https://boringssl.googlesource.com/boringssl/+/8c75ed046f799f1d8b805036b1dea9c5ec0a0fb5%5E%21/#F0 * Bump BoringSSL and fix opaque EVP As OpenSSL, BoringSSL made many types opaque, so it will require updating some of the code to not use fields. * Bump BoringSSL again and fix RSA The same issue - RSA type became opaque, so we need to use accessors similar to what Openssl had. * Bump BoringSSL once more This is (hoperfully) the last bump. This time without issues but we will see what CI says. * Make bignum_to_bytes accept const bignum* It will prevent some of the warnings. This function doesn't mutate bignum anyway. * Update changelog * boringssl: Bump once again * msys2: Update hashes temporarily This are test values because we will move the tag. But for now, let's just test it. * phpthemis: Update version for the sake of testing They will fail probably, but just out of curiosity let's try to run the tests. * Update date of the release Solstice! --------- Co-authored-by: vixentael <vixentael@users.noreply.github.com> * msys2: Update checksums (#1012) It's the egg-chicken problem: we can update those hashes only after the release. But then, the release tag will not point to the updated hashes. * Squashed commit of the following: commit 784033b Author: Anatolii Lishchynskyi <anatolii.lishchynskyi@cossacklabs.com> Date: Tue Nov 28 21:45:12 2023 +0200 New methods of building/installing PyThemis (#1023) Refactor Makefile: * new target pythemis_make_wheel to create a .whl Python package, current modern format to be installed into virtual environmants * new target pythemis_install_wheel to install it in currently active virtualenv * new target deb_python that builds a .deb package for system-wide installation of PyThemis * new target pythemis_install_deb, alias for pythemis_deb + apt install of the created pkg * new target rpm_python, similar to deb_python * new target pythemis_install_rpm, pythemis_install_deb Update GitHub Actions workflow, test .whl and .deb installation See cossacklabs/product-docs/pull/317 for related docs update commit 30578c8 Author: Anatolii Lishchynskyi <anatolii.lishchynskyi@cossacklabs.com> Date: Mon Nov 27 17:53:02 2023 +0200 Fix clippy and fmt issues, update MSRV (#1039) Fix clippy and fmt issues Update MSRV to 1.60 Freeze test deps so they compile on Rust 1.60, with no effect on themis itself (does not use those frozen crates) commit 6111766 Author: Anatolii Lishchynskyi <anatolii.lishchynskyi@cossacklabs.com> Date: Tue Nov 21 17:08:51 2023 +0200 Update emscripten requirements and WASM CI job (#1036) * Bump emsdk version to 3.1.47 Produces module importable in Node v18 With older (emsdk 3.0.0) version, generated `libthemis.js` that should load `libthemis.wasm` fails due to some internal autogenerated code working differently on v16 and v18 * Add link flag for WASM builds Needed to make new emscripten produce working module * Update Node testing versions Add v18 that should now work, also add v20 that is in active development as of now, but if tests are green then why not? * Switched integration tests to use v16 * Removed testing of quite old and deprecated v10 * Updated version of BoringSSL submodule to a newer one (not the latest though) commit 05cac26 Author: Anatolii Lishchynskyi <anatolii.lishchynskyi@cossacklabs.com> Date: Thu Nov 9 22:24:35 2023 +0200 CI fixes (#1034) Rix Rust CI builds (#1032) * Pin log dependency to 0.4.17 * Pin byteorder dependency to 1.4.3 Last versions that still work with current MSRV 1.58. Fix sanitizers CI job (#1033) * Switch to GCC 10 * Install `libgcc-10-dev` that provides file `libtsan_preinit.o` needed for thread sanitizer Change priority of PREFIX in CFLAGS and LDFLAGS (#1031) * Change include dir priority in CFLAGS Move 'CFLAGS += -I/usr/local/include' below engine selection macros, so that if OS have OpenSSL header files installed in /usr/local/include, it won't mess with vendored BoringSSL or any other selected engine. * Put /usr/local/lib in separate LDFLAGS Introduce ADDITIONAL_LDFLAGS macro and put it after both LDFLAGS and CRYPTO_ENGINE_LDFLAGS during linking. * Add optional verbose logging to Makefile Add VERBOSE option to Makefile. If set, print executed command (compiler, linker etc) even for successful runs, and run ldd on created shared libraries. * Enable verbose builds on macOS jobs commit 06d52f4 Author: Anatolii Lishchynskyi <anatolii.lishchynskyi@cossacklabs.com> Date: Tue Sep 26 13:15:37 2023 +0300 Fix make target rbthemis_uninstall (#1022) * Fix make target rbthemis_uninstall * Add RbThemis uninstall step to CI commit 8609650 Author: Nazar Serhiichuk <43041209+G1gg1L3s@users.noreply.github.com> Date: Mon Jul 10 22:57:46 2023 +0300 Avoid specific versions in README (#1016) Just to avoid responsibility of updating it during release (•_•) ( •_•)>⌐■-■ (⌐■_■) commit 3219654 Author: Martin Arista <hello@mrtnrst.com> Date: Mon Jul 10 11:54:25 2023 -0400 Update README.md (#1015) update links for maven and java/kotlin links * Bump boringssl * Revert "Bump boringssl" This reverts commit dee8388. Okay, this time I checked, the previous commit of boiringssl was newer than one in the master. * msys2: Fix hashes for 0.15.0 release (#1040) Since we already have 0.15.0 assets on GitHub, we can specify those and expect successful builds on stable. --------- Co-authored-by: vixentael <vixentael@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a part of T2779. In python world the setup.py is considered deprecated and developers are encouraged to use
pyproject.toml. So, let's try integrating it.I used setuptools as a backend for no particular reasons because the name is familiar and we have no reasons to not use it or use something else.
Also, I tried to test it with newer version of python. After many hours I didn't find any problems with our new approach - pythemis works at least with python 3.4-3.11. Starting from 3.3 I encountered some issues so I didn't bothered to fix them.
However, python3.5 is deprecated and it produces warning like "DEPRECATION: Python 3.5 reached the end of its life on..." so many Python libraries don't support it. , so I decided to remove support of python<3.6 in our metafiles.
Though, actually python3.6 is also deprecated. The same will be true for python3.7 in a couple of days (Jun 27 2023).
So the question is should we declare support of these versions? Or should we declare support of the whole range python 3.4-3.11? We could do that, but we don't really test the whole range. Even our CI is based mostly on testing using different distros which come with different python versions. Still, I will do as you will say.
Checklist
Benchmark results are attached (if applicable)Public API has proper documentationExample projects and code samples are up-to-date (in case of API changes)