Fix provenance of NCL figures created using the log_provenance function

[bouweandela]

Description

Logging provenance using the plot_file entry of the corresponding NetCDF file is a buggy feature of ESMValCore. It is also not supported by the functionality that creates the index.html page. This pull request improves the NCL log_provenance function so it writes a separate entry for image files and they are nice shown in the resulting webpage.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[schlunma]

General question on this before the review to make sure we are on the same page. What do you mean by

Logging provenance using the plot_file entry of the corresponding NetCDF file is a buggy feature of ESMValCore.

? I always use the attribute plot_file in my (python) diagnostics as it was recommended by an early version of the python example diagnostic. In addition, also the documentation advertises the use of plot_file. I'm also pretty sure that the intended way (at least in earlier times) was to always write a netCDF file for a plot and add the provenance to this. So what is the desired way to implement provenance?

• Write it to the .nc file and add the corresponding plot with plot_file, i.e.

provenance_logger['plot_file`] = 'path_to_plot.png'
with ProvenanceLogger(cfg) as provenance_logger:
      provenance_logger.log('path_to_dataset.nc', provenance_record)

• Write it to the .png file, i.e.

with ProvenanceLogger(cfg) as provenance_logger:
      provenance_logger.log('path_to_plot.png', provenance_record)

• Write it to both separately

with ProvenanceLogger(cfg) as provenance_logger:
      provenance_logger.log('path_to_dataset.nc', provenance_record)
with ProvenanceLogger(cfg) as provenance_logger:
      provenance_logger.log('path_to_plot.png', provenance_record)

?

[bouweandela]

Good questions @schlunma and my apologies for the confusion. The only working way of writing provenance at the moment is the last option you describe: log the provenance for every file separately. The documentation that describes that lives here: https://docs.esmvaltool.org/en/latest/community/diagnostic.html#provenance-items-provided-by-the-diagnostic-script. Unfortunately, it looks like I forgot to update the documentation that you linked, I'll update that too.

Indeed using plot_file as an attribute was the recommended way to do it until I changed it about a year ago in #1827, but did you ever notice that it does not work? The only thing it does is embed the provenance of the associated NetCDF file in the .png, provided that you happen to select .png as an output format. If you select another output format, e.g. .pdf, no provenance is available at all for the figure. Because embedding provenance is giving our users problems, I'm considering removing the embedded provenance feature completely and only writing the provenance to the _provenance.xml file: ESMValGroup/ESMValCore#1148.

[bouweandela]

I created a pull request with updated documentation here: ESMValGroup/ESMValCore#1305

[schlunma]

Code looks good and (two) test recipes ran successfully 🎉

Before this change, NCL diagnostics only output provenance for the nc files; now, they also output provenance information for the plot files without any change in the diagnostics themselves. Great job!

One question: Would it be possible to implement a similar feature for the python provenance logging to support the "deprecated" interface? I think this would be really helpful. There are so many diagnostic that use this old version (at least 10+ diagnostics that I have written) and I guess most developers (e.g., I) do not have the time to overhaul these diagnostics.

[bouweandela]

I'll have a look at the Python interface and see if it's easy to fit something in.