Need to improve plot generation performance

[nusbaume]

New feature type

New plot and/or plot enhancement, New infrastructure or infrastructure enhancement

What is this new feature?

During the AMP lunch meeting today, it was discussed how the plot generation step is by far the slowest feature in the ADF, and is currently the biggest reason why the ADF isn't as fast as the original AMWG diagnostics. Thus it would be good to see if there are some relatively easy-to-implement solutions that can be done now to try and speed-up that process, such as generating raster maps instead of the standard contour plots, or parallelizing the plotting scripts themselves similar to what is done for time series and climatology generation.

Assistance required?

No, I will make a PR when the feature is ready

Extra info

Any ideas are welcome! Just comment on this issue if you have anything to add. Thanks!

[justin-richling]

For the polar plots, would it be worth looking into gdal or ncremap or other command line language to re-project the data before it gets to the plots? It seems we lose some time when re-projecting as the polar plots are being made in python.

[brianpm]

Do we have data to support the idea that plot generation is slower than NCL? I don't have a good feeling for what makes plotting fast or not. For contours and transforms, there could be actual algorithmic differences. I'm not sure if GDAL would save us from any of those calculations (I don't really understand GDAL). If all the time is being spent rendering (as opposed to calculating), I wonder whether there is a backend issue that could be customized to be faster (could be architecture dependent?)?

A couple things, just to remember to check:

• https://matplotlib.org/stable/users/explain/performance.html
• https://scitools.org.uk/cartopy/docs/latest/ notes we might try PYPROJ_GLOBAL_CONTEXT=ON

Also, if we really get into this, we can explore non-Matplotlib options (e.g., plotly), but I'm not sure whether the mapping capabilities are as good.