Recovery when release automation fails

[itowlson]

A user hit a problem where Spin 1.3 could not compile a newly created Rust app. This turned out to be because they were using HEAD templates, which were temporarily broken (a conscious decision whose reasons we need to address but are out of scope for this issue). This in turn was because the release automation had not created the spin/templates/v1.3. Which was because the template tagging automation is triggered only by a PR from fermybot, and for 1.3 the relevant PR was sent by a human. Which I believe was because something went wrong in the automation that was meant to update the template SDK references, and human intervention was needed to get things back on track.

(Don't ask me for five whys. That's all the whys I got.)

How should we avoid this happening in future? E.g. some or all of:

• More comprehensive documentation on recovery steps when automation fails
• More flexible automation, able to accept manual interventions
• Being able to manually force the automation to run (from wherever it failed)
• Post-release checks that all artifacts, tags, etc. exist