Switch to cross-compilation for aarch64/ppc64le

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-webservice.

I've rerendered the recipe as instructed in #196.

Here's a checklist to do before merging.

• Bump the build number if needed.

Fixes #196

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[leofang]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you but ran into some issues. Please check the output logs of the latest rerendering GitHub actions workflow run for errors. You can also ping conda-forge/core for further assistance or try re-rendering locally.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cupy-feedstock/actions/runs/4684900434.

[leofang]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cupy-feedstock/actions/runs/4696142986.

[leofang]

@conda-forge-admin, please restart ci

[leofang]

It seems the build_and_run function in CuPy's build system does not allow cross compilation. Not sure if there exists any easy patch.
https://github.com/cupy/cupy/blob/d6252bed4a16aa606681ed97a4fafba2d728dfdb/install/cupy_builder/install_build.py#L717-L748

[leofang]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cupy-feedstock/actions/runs/4702805171.

[leofang]

It seems the build_and_run function in CuPy's build system does not allow cross compilation. Not sure if there exists any easy patch.
https://github.com/cupy/cupy/blob/d6252bed4a16aa606681ed97a4fafba2d728dfdb/install/cupy_builder/install_build.py#L717-L748

@conda-forge/cupy I think everything works now. I've turned on artifact uploading, will ask RAPIDS folks for help testing next week. While waiting for CI to finish, let me explain my finding.

We need to patch build_and_run() as explained above. Ultimately, it needs to run in the build environment (in this case, linux-64), not the target environment (aarch64 or ppc64le). So, we need to use the right compiler, and not let CF's cross-python kick in to modify the compiler/linker flags, which was the first place I struggled the most. I ended up creating a fresh ccompiler object and only using it inside preconfigure_modules().

We happen to also need to patch build_shlib() only because both functions are called inside preconfigure_modules() and they see the same ccompiler object. We don't have to use the same ccompiler but I prefer to keep the patch simple.
This is another place where I struggled with obscure error msgs.

We still need to use the CUDA 11.2 docker image to provide headers/stub libraries for the "build" (linux64), or native compiler. No need to change $CUDA_PATH (which points to /usr/local/cuda in the container). By doing so, we match the CUDA environment seen by both build/host environments.

Finally, when the Python extension modules are built, the "host" (aarch64/ppc64le), or cross, compiler kicks in. cross-python simply does the right job for us (I hope!)🤞

The caveat of this PR is that CF is currently set up to only support cross-compiling with CUDA 11.2 (and CUDA 12 once it's done). CUDA 11.0/11.1 users need to update. (Though v12.0.0 build number 0 does already support CUDA 11.0/11.1.) But given the relatively low download count on CUDA 11.0/11.1 + aarch64 (and we already disabled them for ppc64le) and that RAPIDS is on the latest CUDA 11.x, I don't think this is a concern.

Since building CuPy for aarch64/ppc64le is awfully time consuming using QEMU, and the GPU CI runner only offers the x86-64 platform, I strongly prefer to get this PR merged once we confirm it works. It also helps save the free CI resources and serves good intent for the whole community.

[leofang]

@conda-forge-admin, please rerender

[leofang]

I've turned on artifact uploading, will ask RAPIDS folks for help testing next week.

I forgot that I have access to aarch64 nodes, I just tested locally and everything looks fine. Will merge this by EOB Monday to allow time for feedbacks.

[leofang]

@conda-forge-admin, please rerender

[leofang]

Changed the target branch to cuda-11 since I'll be preparing for the CUDA 12 work (#199) and this is independent of that. Set automerge.

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cupy-feedstock/actions/runs/4728001858.

[github-actions]

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

• linter: passed
• azure: passed

Thus the PR was passing and merged! Have a great day!