Performance scaling of LIS on HPC

[bhanu-magotra]

Hi,
I need suggestions to why LIS is not gaining performance speedup beyond 64 cores (8x8) for my experiments. I have ran following experiments using same LIS config (coldstart,NoahMP+HyMAP2, MERRA2 Forcing, South-Asia Domain, 0.1 Degrees with 15 minutes timestep and Daily output, 1st Jan 2020 to 1st Feb 2020) :

The HPC has Skylake nodes (40cpus, 96GB RAM per node).
Please suggest how I can achieve performance scaling on the HPC or what could be the possible bottleneck reasons.
I am also attaching the lis.config and a sample PBS job script for reference.
LIS_BMARK_config.txt
LIS_BMARK_PBS.txt

[agetiran]

Hi @bhanu-magotra,
this is an existing issue related to the HyMAP parallelization, which is not fully optimized for scalability.

While we don't find a solution for it, I recommend that you use the configuration that gives you the lowest wall time. Alternatively, you can try to run the LSM, then HyMAP in offline mode using LSM outputs as forcings.

Augusto

[bhanu-magotra]

Thanks @agetiran . We could see the expected performance after turning off HyMAP.

[msaharia]

Thanks @agetiran. We might separate the LSM runs and HyMAP in offline mode for historical runs in the future.