Fix cesm link step in builds with high pcols values

[gdicker1]

In GPU runs of CAM, it is desirable to increase the number of physics columns each MPI rank is responsible for by increasing the pcols variable. When pcols grows somewhere beyond 2048, builds fail in the link step.

Adding -mcmodel=medium to the FFLAGS and LDFLAGS allow builds to proceed. Since this can have performance implications, this is only being applied to the NVHPC compilers (the main compilers for GPU applications right now).

[gdicker1]

@sjsprecious, apologies if you aren't the right person to direct questions or this PR to.

I added you as a reviewer for your opinion on this change. Should this be made even more GPU specific? Or, should I apply this to intel and gnu compilers too? Any other thoughts?

[gdicker1]

Once incorporated, this will address EarthWorksOrg/EarthWorks Issue #56

[sjsprecious]

Hi @gdicker1 , my understanding is that the -mcmodel=medium flag is only needed when you want to set a large PCOLS value. According to ChatGPT:

Using the -mcmodel=medium flag can hurt performance compared to the default -mcmodel=small because it requires the compiler to generate larger and potentially less efficient instructions to access global and static variables. This can lead to increased instruction cache pressure and slower memory accesses.

Therefore, I prefer not to set this flag by default and let a user add it manually when needed. Even for a GPU run, we may not have a large PCOLS value depending on the compset, resolution, number of CPU cores / GPUs per node, etc.

On the other hand, using a large PCOLS value will unfortunately hurt the CPU performance. Thus using the default PCOLS value makes sense for any CPU simulation and we should not need to add this flag for GNU/Intel compiler.

This is just my two cents.

[gdicker1]

Thanks for the perspective @sjsprecious! I'll consider this a bit more...

You're correct that we only need to consider adding this flag for higher pcols values. Right now -mcmodel=medium might be something we want for GPU runs, based on Supreeth's work with 1 MPI rank per GPU (if I remember correctly). It seems like a cumbersome step to expect users to add it themselves (but GPU runs already have a few "special" steps).

[gdicker1]

@supreethms1809 could you offer some thoughts or review?

I just updated this so -mcmodel=medium is only used for OpenACC offloads and the changes are at the bottom of the file below a "EarthWorks specifc: ..." comment.

[supreethms1809]

From my limited tests on CPUs, I haven't seen the flag hurting the performance. But it is needed for GPU builds.

@areanddee Do we expect to run with bigger PCOLs on CPUs as well?

[areanddee]

Hi @gdicker1 , my understanding is that the -mcmodel=medium flag is only needed when you want to set a large PCOLS value. According to ChatGPT:

Using the -mcmodel=medium flag can hurt performance compared to the default -mcmodel=small because it requires the compiler to generate larger and potentially less efficient instructions to access global and static variables. This can lead to increased instruction cache pressure and slower memory accesses.

Therefore, I prefer not to set this flag by default and let a user add it manually when needed. Even for a GPU run, we may not have a large PCOLS value depending on the compset, resolution, number of CPU cores / GPUs per node, etc.

On the other hand, using a large PCOLS value will unfortunately hurt the CPU performance. Thus using the default PCOLS value makes sense for any CPU simulation and we should not need to add this flag for GNU/Intel compiler.

This is just my two cents.

My 2 cents:
I would recommend we set this by default ON for GPUs and OFF for CPUs. We already have flags that are specifically for GPUs so I don't think this sets any sort of bad precedent.

[gdicker1]

Thanks for your thoughts @areanddee. Since this will only activate for GPU builds (and more specifically OpenACC builds), I think this is good to move forward.

[gdicker1]

Force push from c3bc0bd to b8a71e2 was rebasing this branch from being based on ESMCI/ccs_config_cesm tag ccs_config_cesm0.0.99 to being based on ew-develop (currently ccs_config-ew2.2.000).