Are the displacement seismograms generated by the solver expected to differ with number of GPUs/compute nodes?

[frawanye]

Hi all,

I'm running a specfem3d cartesian strong scaling benchmark. The benchmark has a fixed reference of the displacements, and computes a) the max absolute error in displacements at the receivers, and b) the L2 error for the full displacement difference vector at the receivers across timesteps. I'm noticing that as I increase the number of GPUs, particularly from 8-16 (which in our case means going from 1 node to 2 nodes), the max absolute error and the L2 error increases substantially.

The max absolute error goes from 4.2-5.0e-6 to 1.6e-5, while the L2 error goes from 6.0-6.1e-4 to 2.8e-3. Technically, these errors seem rather small, but the fact that they are growing is causing some concern. All of these runs use the same workload size (~1.4M elements running for 48,000 time steps, NEX_XI=256, NEX_ETA=256). All runs complete cleanly, it's just the error that changes.

Does anyone know if this is expected behavior, or is the error expected to be roughly the same regardless of the number of GPUs and nodes?

For reference, here is a table with the decompositions used and the errors recorded.

Environment: SPECFEM3D v4.1.1-305-gff5ca920 (devel), ./configure --with-mpi --with-hip, AMD GPUs, ROCm 7.2, OpenMPI 5.0.9, single MPI rank per GPU.

Thanks in advance! Happy to provide more information if I'm free to share it, and/or perform more runs to help narrow things down.