CI publishing pipeline

[WrathfulSpatula]

It would be great to have a continuous integration (CI) build and publishing pipeline. (The primitiveness of my current method should rightly embarrass me, but "I know why I do it my way.") pyqrack might require something a little particular for its wheel variants, which is to build one (or two) binary distributions respective to the platforms of its available wheels, and just package that one binary variant in each respective wheel, where the Python code can find the right Qrack binary paths also based upon its own access to system information.

We want each wheel to have a respective Qrack binary build on the same platform as the wheel tag, and only that respective binary, with many respective wheel with binary pairings. I'd love and deeply appreciate help from the community, on this one, for I do spend too much time manually building to produce such a result.

[golanor]

I'm commenting here just to remind myself to take a look at this after unitary hack is over. It's been fun working on the testing PR, and I've seen more places I could help.

[WrathfulSpatula]

Thank you for your contribution, and it's great to hear you're considering making another! Let me know, if I can help in any way!

[WrathfulSpatula]

By the way, this was generated by an LLM, (with credit given to them,) but it might help this issue:
https://github.com/unitaryfund/pyqrack/blob/pypi_package/package.sh

[anushkrishnav]

@WrathfulSpatula I can take this issue, has anything changed from the requirement ? Please assign me this, will work with you on whats expected

[golanor]

Hi @WrathfulSpatula , apparently I need the incentive of unitaryHack to contribute code 😅

Does the workflow look OK?

[WrathfulSpatula]

@anushkrishnav, @golanor, I'm just starting my day at work, here. Anush, unfortunately, as part of the hackathon event design is that it's a "race" to get the first acceptable PR merged, and since Or worked with the repository in last year's hackathon and has a PR already up, I'll let you know it's likely I'm going to accept the PR that's already on the repo, once I test it and iterate on it. Sorry that we can't accept two solutions, since the repository only needs one, but I wanted to let you both know so you had more opportunity to look elsewhere for bounties.

It's worth mentioning, there are other Qrack bounties, all valued at $125 apiece:
https://unitaryhack.dev/projects/qrack/

I also develop https://metriq.info with Vincent Russo, at Unitary Fund, and these might be some of the only React.js and web development issues in the hackathon:
https://unitaryhack.dev/projects/metriq/

Sorry I can't award the bounty twice!

@golanor, it looks good, but I need to test that the artifacts it produces actually work. I'm looking into it, but let me know if you could provide a bit more detail on how I should build and retrieve the artifacts. We might need certain additional considerations, like adding the OpenCL library package to the build environment so the wheels are built with GPU acceleration (even though the CD build environment doesn't have and doesn't need a GPU, absolutely fine, despite building the wheels with OpenCL support, which should just work based upon the presence of ocl-icd-opencl-dev and opencl-headers Ubuntu packages on build).

[WrathfulSpatula]

If there's significant additional work to do, you two might consider working together and splitting the bounty.

Or, as per how PyQrack is currently packaged, I'd kind of like to do it how the packaging script does it, where only the binary artifacts relevant to the particular platform are included, such that we bring the size per wheel down from something like 30-ish MB across the board to more like roughly 8 MB per, give-or-take.

In fact, I think your workflow misses the point a bit, still. We need to do something like building the (C++) Qrack artifact in the CD environment, then packaging it with just its specific respective wheel, per platform.

Also, we will need OpenCL support, which should work automatically just if Qrack is built in the presence of the ocl-icd-opencl-dev and opencl-headers Ubuntu packages.

[WrathfulSpatula]

@anushkrishnav @golanor - Sorry for the half-day delay in getting you sufficient clarification on the issue.

@golanor, your approach is one the right track, but here's how I've packaged PyQrack, to date: I've locally built (C++) Qrack library artifacts on every permutation of operating system with processor instruction set that I own, basically. Each "artifact," per platform, goes into a different Python wheel. Or, your current PR builds the wheel on different platforms, but it also needs to build C++ Qrack on different platforms.

One relatively simple way to achieve this involves basically just adding the C++ Qrack library as a git submodule. During the packaging pipeline, you build Qrack on whatever the local platform is, so that the Qrack library files can be used for that operating system and CPU instruction set.

qrack_system.py controls the paths PyQrack checks to look for the C++ Qrack library files. You could change the constructor to look in a new Qrack submodule. Then, just automate the building and linking of Qrack in the GitHub workflow you have, @golanor.

(You two could decide to work together to split the bounty. That's up to both of you, and either might choose to prefer to compete. For now, the bounty is up-for-grabs.)

@golanor, your PR is on the right track but incomplete, and we'll discuss OpenCL dependencies when someone gets there, if necessary, but it's just a matter of installing those two packages I mentioned above, in Ubuntu.)

[WrathfulSpatula]

(Slight clarification: using a "shared library" with a C interface from Python doesn't require a "linking" step. You just build the artifacts and put them where the QrackSystem constructor can find them, and that should be all that's required.)

[golanor]

Hey, I think I'm missing something. After I run cmake on the qrack repository, how do I get the shared library that I'm supposed to use for pyqrack? I can't find any .so file in my build folder after running the cmake instruction from the readme.

[anushkrishnav]

@anushkrishnav @golanor - Sorry for the half-day delay in getting you sufficient clarification on the issue.

@golanor, your approach is one the right track, but here's how I've packaged PyQrack, to date: I've locally built (C++) Qrack library artifacts on every permutation of operating system with processor instruction set that I own, basically. Each "artifact," per platform, goes into a different Python wheel. Or, your current PR builds the wheel on different platforms, but it also needs to build C++ Qrack on different platforms.

One relatively simple way to achieve this involves basically just adding the C++ Qrack library as a git submodule. During the packaging pipeline, you build Qrack on whatever the local platform is, so that the Qrack library files can be used for that operating system and CPU instruction set.

qrack_system.py controls the paths PyQrack checks to look for the C++ Qrack library files. You could change the constructor to look in a new Qrack submodule. Then, just automate the building and linking of Qrack in the GitHub workflow you have, @golanor.

(You two could decide to work together to split the bounty. That's up to both of you, and either might choose to prefer to compete. For now, the bounty is up-for-grabs.)

@golanor, your PR is on the right track but incomplete, and we'll discuss OpenCL dependencies when someone gets there, if necessary, but it's just a matter of installing those two packages I mentioned above, in Ubuntu.)

Its alright, I understand , @golanor good going

[WrathfulSpatula]

Hey, I think I'm missing something. After I run cmake on the qrack repository, how do I get the shared library that I'm supposed to use for pyqrack? I can't find any .so file in my build folder after running the cmake instruction from the readme.

@golanor After cmake, most C++ projects still need to make all. (You can make all -j16 for a faster build, if you have 16 CPU hyperthreads, or replace the number with however many you have.)

[golanor]

When I try to make with CUDA enabled, I get the following error:

make[2]: *** [CMakeFiles/qrack.dir/build.make:455: CMakeFiles/qrack.dir/src/common/cudaengine.cu.o] Error 1

Couldn't find an issue on this - any idea what can cause this?

[WrathfulSpatula]

When I try to make with CUDA enabled, I get the following error:

make[2]: *** [CMakeFiles/qrack.dir/build.make:455: CMakeFiles/qrack.dir/src/common/cudaengine.cu.o] Error 1

Couldn't find an issue on this - any idea what can cause this?

We don't actually package CUDA builds by default for PyQrack, but it builds fine on my local machine. The problem might be lack of an up-to-date CUDA toolkit or an ambiguous CUDA architecture. The build tools should attempt to automatically detect your CUDA architectures, but this can fail, also depending on CUDA toolkit version. I forget what the command line option is, but you can manually specify your CUDA architectures, alternatively.

(FYI, no one believes me, but it's only because of NVIDIA's marketing department: on NVIDIA hardware, OpenCL is nonetheless the faster option of the two, while the two APIs and implementations are nearly one-to-one and the CUDA profiler reports no bottlenecks. The only potential application we've found for which we might want CUDA is for their proprietary NVLink interconnect hardware, and that might turn out to be a marginal improvement over PCIe, given the performance penalty for CUDA vs. OpenCL in general.)

[WrathfulSpatula]

Actually, the option is -DQRACK_CUDA_ARCHITECTURES=[x], but [x] is defined by CUDA:
https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

[golanor]

I'm trying to build qrack on my system to understand the structure of the build (although I did still spam github actions). The github workflow builds Qrack but I still need to point to the correct file and add it to the PyQrack wheel.

When building on my system:

1. Cuda version 12.5.40 fails with the error above.
2. OpenCL - I have Nvidia OpenCL 550.78-1.
   running:

cmake -DENABLE_RDRAND=OFF -DENABLE_DEVRAND=ON -DQBCAPPOW=12 -DCPP_STD=14 -DUINTPOW=5 ..

Fails with:

In file included from /home/or/Code/python/open_source/qrack/include/qpager.hpp:16,
                 from /home/or/Code/python/open_source/qrack/include/qfactory.hpp:16,
                 from /home/or/Code/python/open_source/qrack/src/qunit.cpp:24:
/home/or/Code/python/open_source/qrack/include/common/oclengine.hpp:40:10: fatal error: CL/cl2.hpp: No such file or directory
   40 | #include <CL/cl2.hpp>
      |          ^~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/qrack.dir/build.make:288: CMakeFiles/qrack.dir/src/qunit.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:142: CMakeFiles/qrack.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

[WrathfulSpatula]

2. OpenCL - I have Nvidia OpenCL 550.78-1.
   running...

You don't have the OpenCL headers, it looks like. (They're installed separately.) If you're on Ubuntu, for example, sudo apt install opencl-headers.

[golanor]

OK, I managed to build qrack (also on my system), and place the resulting libqrack_pinvoke.so both in pyqrack/qrack_system/qrack_lib/ and pointed at by $PYQRACK_SHARED_LIB_PATH, but it isn't added to the final wheel. Any idea why not?
@WrathfulSpatula

[WrathfulSpatula]

OK, I managed to build qrack (also on my system), and place the resulting libqrack_pinvoke.so both in pyqrack/qrack_system/qrack_lib/ and pointed at by $PYQRACK_SHARED_LIB_PATH, but it isn't added to the final wheel. Any idea why not?

MANIFEST.in likely controls this. You might also need to declare the object file dependency in setup.py, but you definitely need to register it in MANIFEST.in.

[golanor]

The build process works for POSIX. Windows seems more complicated - it says in the readme for Qrack that you need to build it in visual studio. Is there a way to do it in the command line?

[WrathfulSpatula]

The build process works for POSIX. Windows seems more complicated - it says in the readme for Qrack that you need to build it in visual studio. Is there a way to do it in the command line?

I actually don't know. However, if you were to finish Linux (and possibly Mac), that might be sufficient for the issue. If I need to build a Windows wheel separately, that would still be a major improvement in the workflow.

[WrathfulSpatula]

@golanor You did major work on this, so you should get the bounty. I'll review later today, but we can amend what you have any way we need. (Thank you!)