/azp run

For production runs on CPUs, it might be better to avoid the Cray compiler altogether and use gcc instead.

Ugh, using fno-cray seems to give a linker error while compiling on Setonix CPUs.

@chongchonghe got it to work on Setonix -- maybe @chongchonghe can help?

On top of the setonix-gpu.profile, I have to add -ffp=0 to both CFLAGS= and CXXFLAGS= and the code compiles and produces correct results for some pure hydro simulations. The radiation runs failed.

On top of the setonix-gpu.profile, I have to add -ffp=0 to both CFLAGS= and CXXFLAGS= and the code compiles and produces correct results for some pure hydro simulations. The radiation runs failed.

But it works if you add -fno-cray, right?

On top of the setonix-gpu.profile, I have to add -ffp=0 to both CFLAGS= and CXXFLAGS= and the code compiles and produces correct results for some pure hydro simulations. The radiation runs failed.

But it works if you add -fno-cray, right?

If I add -fno-cray, the job failed with segfault when running with MPI.

So then fno-cray is not going to work for all the test problems? @BenWibking

So then fno-cray is not going to work for all the test problems? @BenWibking

It should be possible to avoid the Cray compiler entirely on Setonix. Then no special options should be necessary.

This should be possible by loading PrgEnv-gnu and then setting the compiler to hipcc:

module load PrgEnv-gnu
export CC=$(which hipcc)
export CXX=$(which hipcc)

So then fno-cray is not going to work for all the test problems? @BenWibking

It should be possible to avoid the Cray compiler entirely on Setonix. Then no special options should be necessary.

This should be possible by loading PrgEnv-gnu and then setting the compiler to hipcc:

module load PrgEnv-gnu
export CC=$(which hipcc)
export CXX=$(which hipcc)

This by itself does not seem to be sufficient. We still need HDF5 and ROCm for example. When I load HDF5 and ROCm modules and make a test problem, I get these warnings:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

I still have some cray modules loaded. Is this an issue:

psharda@setonix-06:~> module list

Currently Loaded Modules:
  1) craype-x86-milan     4) perftools-base/23.03.0                  7) gcc/12.2.0        10) cray-mpich/8.1.25      13) pawsey         16) python/3.10.10
  2) libfabric/1.15.2.0   5) xpmem/2.5.2-2.4_3.47__gd0f7936.shasta   8) craype/2.7.20     11) cray-libsci/23.02.1.1  14) pawseytools
  3) craype-network-ofi   6) pawseyenv/2023.08                       9) cray-dsmml/0.2.2  12) PrgEnv-gnu/8.3.3       15) slurm/22.05.2

I think this is a better solution: #566.

Since there is no activity on this for the past 2 months, I am going to close this issue for now. If it turns out to be needed, we can re-open it.