Automate Zenodo archiving

[jdangerx]

This issue tracks all the tasks required to get our automated archiving system up and running.

Scope

• Archivers create a report describing what happened during an archiving run.
• Each archiver can specify dataset-specific success/failure criteria.
• Archivers are robust to transient networking issues (false "failures" that are external to the data).
• Scheduled archiving runs produce a persistent record of their outcome (probably a GitHub issue).
• Scheduled archiving runs notify us of their outcome (Slack notification + issue assignment).
• All working archivers are being run on a schedule, publishing new archives to the production Zenodo site.

Out of Scope

• Rewrite EPA CEMS archiver to use API #68
• Archive FERC EQR #31
• EQR archiver concurrency/disk limits #44
• FERC EQR Archiver #43

Tasks

Beta Give feedback

1. Update EIA Thermoelectric Cooling Water link matching patterns #77
   eiawater inframundo +2
   
2. Refresh Zenodo Archives Manually #79
   17 of 17
   inframundo zenodo +2
   
3. Add production credentials to workflow
4. Ensure each XBRL filing in FERC RSS feed has a stable ID #73
   bug ferc1 ferc2 ferc6 ferc60 ferc714 inframundo xbrl +8
5. Running archiver with --dry-run leaves an unpublished draft #75
   inframundo zenodo +2
6. Add all new archives to the Catalyst Cooperative community #76
   inframundo zenodo +2
   
7. Increase archiver robustness to transient network issues #74
   0 of 3
   inframundo +1
8. Summarize archiver run and send notification via GH Action #60
   inframundo +1
   
9. Specify archiver success & failure conditions #70
   1 of 17
   Epic inframundo +2
10. Update keywords in new dataset metadata #80
    5 of 5
    eia176 eiawater inframundo metadata phmsagas +5
    
11. "ValueError: I/O operation on closed file" uploading to Zenodo #81
    inframundo mshamines zenodo +3
12. Add validation test that looks at file size #144
    +0
13. Automate remaining archive runs #276
    automation github_actions +2

[e-belfer]

@zaneselvans Do we feel satisfied closing this? We've changed the scope to move away from per-archive conditions since we originally wrote this, but I've closed #276

[zaneselvans]

It seems like the base level "automate" is done, and we could probably gather together various other issues that improve the validation of archives and any archiving bugs we run into.

Can we close the bugs that are listed here now, since they don't seem to have been causing issues? I guess that's just #74 & #81. We should figure out how we want to deal with drafts and continue to make the success/failure conditions more robust.