Jorgen's addition of HIP/CUDA abstraction to codegen and all processes (with extra fixes and tests: "jt774" replacing PR #774)

[valassi]

Hi @oliviermattelaer @roiser @nscottnichols as discussed this is the WIP PR replacing Jorgen's PR #774.

It includes all of Jorgen's changes for HIP in PR #774 (GpuAbstraction.h and hipcc build instructions) in CODEGEN, plus a merge of the current upstream/master and extra fixes that were necessary to fix codegen and/or CUDA/C++ builds. Those extra fixes were partly derived from earlier work I had done with Jorgen in July/August (that I kept for reference in a PR #800, that I opened and immediately closed).

This PR has a fully functioning CODEGEN and regerenated processes, with baisc builds tested for ggtt.mad, but it is still in WIP.

• To start with, I need to run all cuda/c++ tests (some will run in the CI, the performance suite I will run tonight).
• More importantly, ** I HAVE NOT TESTED ANY HIP YET **, neither the build nor the runtime. I need to install hipcc to test the build. And I need to setup a node with an AMD GPU to test the code.

I will close PR #774 because it is replaced by this one.

[oliviermattelaer]

So for me this simple madgraph test is not working anymore:

generate g g > t t~ g
output madevent_simd PROC_TEST
launch
set cudacpp_backend FORTRAN

crash is

Command "import /Users/omattelaer/Documents/git_workspace/3.1.1_lo_vectorization/cmd_multi" interrupted in sub-command:
"output madevent_simd PROC_TEST" with error:
AttributeError : 'SIMD_ProcessExporter' object has no attribute 'export_processes'
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in 'MG5_debug'.
Please attach this file to your report.

Does it work for you andrea?
If yes which of the MG5aMC branch is needed to have this working?
(I bet that this might be my issue)

[oliviermattelaer]

I was using gpucpp_wrap, indeed gpucpp is working. Let me check with that one that SIMD is working now on my mac

[valassi]

Hi @oliviermattelaer thanks yes I am using gpucpp branch. Or more precisely: I am using whatever mg5amcnlo commit we have in the module:
https://github.com/valassi/madgraph4gpu/tree/jt774/MG5aMC
Now this points to
https://github.com/mg5amcnlo/mg5amcnlo/tree/23f61b93fdf268a1cdcbd363cd449c88b3511d7a

This is a rather older version of gpucpp. If you want, I can also try to include the upgrade to the latest gpucpp here. But first check with 23f61b9 please... again, I'd avoid mixing everything together if possible (it's ok if it works, but if it does not work I would rather debug the issues separately). Thanks!

[valassi]

PS 23f61b9 is also what we have now in master
https://github.com/madgraph5/madgraph4gpu/tree/master/MG5aMC

[oliviermattelaer]

I can not test with that MG5aMC version since my python version is too recent for that "old" version of MG5aMC...
So I had to update (both gpucpp and gpucpp_wrap) to merge with 3.5.3...
But seems working so far...

[oliviermattelaer]

Ok, it does not work for me actually.
They are some issue with the python interface of the plugin (nothing related to the hip part).
I guess that such issue is solve in another branch (can not be working like that). Will push here some cherry-pick or other method to fix the issue that I have (and then work on the next one).

[oliviermattelaer]

note for later, we will need to re-run the test on that commit since the reported crash is an github API issue (i.e. not related to the commit per say)

[valassi]

Ok, it does not work for me actually. They are some issue with the python interface of the plugin (nothing related to the hip part). I guess that such issue is solve in another branch (can not be working like that). Will push here some cherry-pick or other method to fix the issue that I have (and then work on the next one).

Hi @oliviermattelaer thanks for testing and for adding the fix 9ed3aaf

As you saw I opened PR #811 in parallel about the update of mg5amcnlo to the latest gpucpp. But I suggest we postpone that, and we use the older gpucpp for this PR #801.

And yes we do too much for github and its CI ;-)

[oliviermattelaer]

I have check the plugin/interface part and this is good to merge for me.

However, this would need to add a new backend within launch_plugin.py
(for the moment, this only support Fortran, CPP, CUDA).
Which means that a "normal" user can not run the code (even if all the script that we use for testing are working).

I would say that our target here would be to be able to run via the following script:

generate g g > t t~ g
output madevent_gpu PROC_TEST
launch
set cudacpp_backend HIP

(maybe a good time to change the name of that variable)
I can implement that within this PR (this is easy for me) but I will not be able to test it (no access to LUMI).
Or do you prefer that we discuss/add that possibility in another branch? (both are fine for me)
If this is the case, then we can move forward for the moment.

[oliviermattelaer]

let me change the flag to approve, since my "requested change" are not strictly necessary for us to merge this (and in that case, I will create a new branch that will implement such change and ask you the check them)

[valassi]

I have check the plugin/interface part and this is good to merge for me.

However, this would need to add a new backend within launch_plugin.py (for the moment, this only support Fortran, CPP, CUDA). Which means that a "normal" user can not run the code (even if all the script that we use for testing are working).

I would say that our target here would be to be able to run via the following script:

generate g g > t t~ g
output madevent_gpu PROC_TEST
launch
set cudacpp_backend HIP

Thanks Olivier, I will merge. I committed a few more manual test logs, all OK. And the CI is good.

After this merge 801, I would suggest going in the following order:

• adapt and merge Jorgen's makefiles with separate builds for CUDA, HIP and C++ #798 about separate cpp, cuda, hip builds (built over Jorgen's Includes all the changes to the makefiles, adding new targets for CUDA/cppavx builds #775)
• adapt and merge remove gXXX.cu symlinks (build XXX_cu.o from XXX.cc) #368, removing gXXX.cu links and using XXX.o and XXX_cu.o instead (and possibly also XXX_hip.o, now)

Also, later on,

• adapt and merge update the mg5amcnlo submodule to the latest commit in branch gpucpp #811 about moving to the latest gpucpp
• adapt and merge add support for ROCRAND (via HIPRAND) #809 about rocrand (this is actually almost ready too, I just realised that RocrandHost cannot be implemented apparently, can be studied in more detail, maybe needs a more recent HIP, but I think that's ok for the moment)

And then later all the other non hip stuff (warp size, more on makefiles, etc).

[valassi]

And thanks @Jooorgen again! :-)