Build and package Catalyst wheels with OpenMP and ZStd

[maliasadi]

We now build the Lightning simulator suite with enabling OpenMP and LLVM deps and dialects with ZStd on Linux and macOS. For ABI compatibility, we package our wheels with these libraries and check the hidden deps using

• auditwheel on Linux
• delocate on macOS

[sc-52635]
[sc-52636]

[josh146]

Thanks @maliasadi! rather than myself reviewing, @mlxd might be a better reviewer here (unless there was something specific you wanted to me check)

[rauletorresc]

If I understand correctly, we don't ship libomp and zstd together with our package, we only force their installation from the software manager of the underlaying operating system. In this case, Linux systems would probably install the GCC version and MacOS systems the LLVM based one. Am I right?

[maliasadi]

we don't ship libomp and zstd together with our package, we only force their installation from the software manager of the underlaying operating system. In this case, Linux systems would probably install the GCC version and MacOS systems the LLVM based one. Am I right?

@rauletorresc We indeed ship Catalyst packages with both libomp and zstd using auditwheel on Linux (and delocate-wheels on macOS) as the result of this PR. auditwheel (and delocate) repair our wheels with copying external shared libraries dependencies into the wheel. These tools also modify the appropriate RPATH entries to load these libraries at runtime. Packaging wheels with the OMP runtime and zstd can be considered as good practices as it will ease the installation of packages for users (without any needs to install zstd and omp as additional system requirements). This also ensures the ABI compatibility with external libs in the process.

[rauletorresc]

we don't ship libomp and zstd together with our package, we only force their installation from the software manager of the underlaying operating system. In this case, Linux systems would probably install the GCC version and MacOS systems the LLVM based one. Am I right?

@rauletorresc We indeed ship Catalyst packages with both libomp and zstd using auditwheel on Linux (and delocate-wheels on macOS) as the result of this PR. auditwheel (and delocate) repair our wheels with copying external shared libraries dependencies into the wheel. These tools also modify the appropriate RPATH entries to load these libraries at runtime. Packaging wheels with the OMP runtime and zstd can be considered as good practices as it will ease the installation of packages for users (without any needs to install zstd and omp as additional system requirements). This also ensures the ABI compatibility with external libs in the process.

Would this not conflict/alter/break any previously installed versions of those two libraries at the user's end?

[dime10]

[sc-52635]
[sc-52636]

[dime10]

Quick question, does the PR turn on OpenMP for lightning.kokkos and turn it off for lightning.qubit? This was the conclusion we reached for sensible defaults talking to Lee.

Note we should also apply those configuration defaults to our make scripts :)

[maliasadi]

Would this not conflict/alter/break any previously installed versions of those two libraries at the user's end?

@rauletorresc good question! it shouldn't conflict or break any previous version as in previous versions of Catalyst we didn't build our wheels with OMP and Zstd. We enabled them in this PR.

[maliasadi]

Quick question, does the PR turn on OpenMP for lightning.kokkos and turn it off for lightning.qubit? This was the conclusion we reached for sensible defaults talking to Lee.

It turns on OMP for both lightning simulators. But for lighnting.qubit, this OMP support is mainly for the computation of adjoint-jacobian in parallel. @mlxd added a separate flag to enable OMP for gates PennyLaneAI/pennylane-lightning#510 that is turned off by default. But, if we don't want to use adjoint-jacobian with OMP, then we may want to consider using LQ_ENABLE_KERNEL_OMP=ON instead for the lightning.qubit simulator. I turn them off as we can always use lightning.kokkos for the OMP support. We can revisit this later 👍

Note we should also apply those configuration defaults to our make scripts :)

Sure, will update Makefiles 🙌