PETSc packages uses mpicc instead of cc as compiler on cray

[wspear]

On cray systems like perlmutter and polaris petsc as built by spack will have the petscvariables file set PCC/CC and other relevant compiler/linker variables to the path to mpicc (e.g. /opt/cray/pe/mpich/8.1.16/ofi/gnu/9.1/bin/mpicc). This works in some cases but cray's mpicc will not enable gpu accelerated mpi. The cc compiler must be specified for this functionality. Currently executables built using PETSc's provided make variables will fail to run unless MPICH GPU support is deactivated.

This configuration situation is caused by
"--with-cc=%s" % self.spec["mpi"].mpicc,
In the mpi variant case in the package. I think when cray is detected the package should instead use the default compiler: "--with-cc=%s" % os.environ["CC"],

@balay @BarrySmith @jedbrown @eugeneswalker

[wspear]

I've learned that using the cc/CC/ftn cray compilers is strongly discouraged by spack. Alternative fixes are being investigated.

[balay]

There are 2 ways of using gpu enabled mpi [I might have used the second one with spack]

https://gitlab.com/petsc/petsc/-/blob/main/config/examples/arch-olcf-crusher.py

• link directly with "-lmpi_gtl_hsa" [or similar for cuda]
• load "craype-accel-amd-gfx90a" [or similar for cuda] - but this also requires --with-openmp=1 [otherwise linking in fortran objs fails]

[balay]

Hm - 'modules' I guess work from CC - but not mpicc. So my recollection of my spack install could be wrong [I might have just tested without gpu enabled mpi]