Parallelize the ensemble surface analysis job

[CatherineThomas-NOAA]

What new functionality do you need?

The esfc job runs global_cycle on the ensemble, which updates the surface analysis. New to v17 with the addition of soil DA, the esfc job also regrids the soil increment from the gaussian to the cubed sphere.

The job script is in bash and calls the regridding script first, followed by the call to global cycle. Within the regridding script itself, the script runs a FOR loop over the members and time levels and running the compiled regridding executable serially in each loop iteration. The executable runs quickly but its doing it for all 80 members sequentially, which adds up. If the regridding executable is run in parallel for each member, it would save us a signifcant amount of time for this task.

Acceptance Criteria

• esfc job performs the regridding for ensemble members at once (or in groups).
• Cycling experiments with this feature reproduce serially regridded results.

Suggest a solution (optional)

No response

[CatherineThomas-NOAA]

@DavidNew-NOAA made updates here: https://github.com/DavidNew-NOAA/global-workflow/tree/feature/multi-regrid

It was tested on Hera in the context of the g-w CI. I will now test this at higher resolution.