Summarize archiver run and send notification via GH Action

[e-belfer]

With the move to automated archiving, there's a need to easily and visibly summarize the outcome of the automated archival run. Communication should summarize the status of each archiver's run, and any critical warnings or errors. We could also use this as a chance to create a public record of what's new in each deposition.

Who/Where to notify?

The hope is that these archives will be a public resource, so producing a persistent public record of the data updates and/or allowing the general public to get access to this information would be good. Some options:

• Send a Slack message to a publicly accessible channel.
• Send an email to a list that outside folks can sign up to.
• Create an issue on GitHub containing the summary and requiring that it be reviewed by someone on our team e.g. “Review new data snapshots for 2023-03-14.”
• Automatically tweet a link to the GitHub issue.

Notification / Summary contents?

• Have a section dedicated to each dataset being archived.
• For each dataset, indicate whether the archiving run was labeled a success or a failure.
• If there was no change from the previous version, resulting in no new archive being created, say so explicitly, and indicate how long it has been since the archive was updated (we should eventually get suspicious if archives that we expect to change don't).
• List any change in the set of data partitions that were archived relative to the previous version.
• List any existing data partitions that were updated and indicate the change in file size.
• Summarize how many partitions were updated (number and % changed)
• Summarize the change in archive size (absolute MB and % change)
• If the archiving run was a failure, indicate what criteria failed and provide context. E.g.
  • List any expected but missing data partitions
  • List any unexpected new data partitions
  • List any files that have an unexpected internal filetype, and the type that was expected
  • List any files that shrank so much that they triggered the failure
• If the archiving run failed for reasons unrelated to the data being archived, indicate that as well. E.g.
  • Unable to connect to the data provider
  • Unable to connect to Zenodo
  • GitHub runner ran out of disk space
  • Archiving run took too long and timed out

[e-belfer]

List any change in the set of data partitions that were archived relative to the previous version.

In #70 this is flagged as a condition for failure, so would this not be redundant? Or do you mean if we change the script to include different partitions between archive runs this should be explicitly noted by our archiver?

Otherwise I think this covers all our bases. This seems a bit lengthy for a slack message so presumably one of the other communication options will make more sense here.

[zaneselvans]

I'm thinking that the archiver summary report (Maybe it's a JSON object?) will be the thing that's evaluated to decide whether the run failed. Some changes in the set of data partitions are fine and expected -- e.g. a new month of eia860m data -- but we'd still want to see that change reflected in the summary. Most kinds of changes would be unexpected and bad though -- say the 1995 California EPA CEMS data disappears, or the name of the PHMSA underground natural gas storage partition changes from underground_natural_gas_storage to undergasstore or something. Then it would be useful to see what the change was for debugging.

I think automatically posting the summary of the run as an issue, assigning someone to review it, and then posting links to the issue on Slack / Twitter / Wherever makes sense. It'll be persistent and public and someone will be explicitly tasked with reviewing it, and it'll show up in a place where we can have some discussion and easily spin off new issues if there's something that needs to be fixed.

[jdangerx]

Scope:

• run summary creates a github issue if there are changes (criteria listed above)
• issue is assigned to a person to look at
• makes a slack post

Next steps

• write a script that summarizes an archiver run if there was a new version published

[zaneselvans]

I think we need to summarize and communicate the status of the run regardless of whether a new version is published. If the run fails, we need to know that and whatever useful error information was collected. Under some circumstances even "no change" is suspicious. E.g. if the monthly EPA CEMS hasn't updated in 60 days, it should get flagged for review.

[e-belfer]

That makes sense to me. If I'm understanding correctly your proposal, @zaneselvans, we publish an issue weekly with the information listed above. Most weeks this should look good and we just close it immediately. I think if that's our plan then ideally we'd name the issue to reflect the status of the run, or at least reflect the success status in the slack message. If key failures are occurring it'd be useful to know to prioritize fixing them.