Thoughts on packaging CipheyCore

[harens]

Following from Ciphey/Ciphey#583, I thought I'd give some thoughts for other package maintainers on how to package Ciphey.

For future readers that might want to do this, I'll document some issues I faced below and how I fixed them. If you need any additional help, I recommend reading any MacPorts PRs that I link for further details on each section (especially if you're packaging for macOS) and feel free to ping me.

TL;DR: CipheyCore can be installed by running the following through MacPorts: (project page)

sudo port install py-cipheycore

(Replace it with py38-cipheycore for python 3.8)

Assuming all goes well, this should install it for your system with no segmentation faults (pre-compiled binaries are also available for some macOS versions). Maintainers might find it useful to document this.

---

For a project as large as Ciphey, it's on surprisingly few package managers. In my opinion, it all comes down to CipheyCore. If there was no CipheyCore, it would be a piece of cake to package Ciphey. To quote @Cyclic3 from Ciphey/Ciphey#197 (comment):

The problem is that the project is built in two stages, with the C++ cmake build creating an autogenerated python module (using SWIG), and then a thin poetry wrapper is used to build the wheel

Although this is all very well documented, it’s incredibly difficult to package. Since it’s not possible for there to be a standard distribution (as it contains a platform-specific library), the method used 99% of the time to sort out python dependencies on loads of package managers is thrown out the window.

Therefore, you will have to write a PKGBUILD/Formula/Portfile/etc. for CipheyCore. It is not enough to set it as an external dependency (similar to what Homebrew does for python libraries).

---

How to build CipheyCore

Fortunately, it is very well documented in the README.

Just a few things to note from a package maintainer's perspective. When building the python3 library, unlike the ciphey maintainers, we're only interested in building the source distribution. This is since it will then contain a library that is built for our platform, and we can then deal with it similar to how your package manager would normally deal with python libraries.

For example, MacPorts' pre-built binaries are built for specific macOS versions, so there's no issue with using the platform-specific library. The same is true when MacPorts builds from source.

How to deal with segmentation faults

I ran into this issue macports/macports-ports#9964, but the fix appears simple.

Even if CipheyCore builds successfully, set the -DCIPHEY_CORE_PYTHON flag to the folder containing the headers

e.g. /opt/local/Library/Frameworks/Python.framework/Versions/3.8

The problem is described very well in Ciphey/Ciphey#238 (comment), but for package maintainers it's likely due to the wrong python version being used to configure CipheyCore.

e.g. If python 3.9 is used to build CipheyCore for python 3.8, this will cause a segmentation fault at runtime.

Getting CipheyCore to build if it's not working

As a package maintainer, you have to think about different versions of your operating system. Just because it builds for your version doesn't mean it will work for all versions of your OS (Trust me I learnt this the hard way macports/macports-ports#9932).

The key is C++20 compiler support. Compilers such as AppleClang 10+, Clang 9+, etc. should hopefully be fine.

Over at MacPorts, we manually specified which compilers worked and which ones don't (and possible fallback compilers). You might have to do something similar in case a user does not have a supported compiler installed.

TypeError at runtime

This one's easy. If you're packaging CipheyCore for Ciphey, make sure to set the default python version to 3.8.

If you're packaging CipheyCore just by itself, you can choose either 3.8 or 3.9.

---

As I said before, I'm more than happy to help anyone who is trying to package Ciphey/CipheyCore. In case you're interested (or stuck), the final Portfile for CipheyCore can be found here. You might find the comments in the Portfile helpful. I really recommend reading the MacPorts PRs linked if you're lost on a particular section, and let me know if you have any questions.

All the best,
Haren