Clarify how to resolve merge conflict during ceremony merge to main

[kommendorkapten]

Description

The ceremony branches have quite strong branch protection rules (linear history and no force pushes). This can create an issue if there is a merge conflict against main, as there is no way to resolve the conflict in a satisfactory way given the branch protection rules (at least not to my knowledge). This happened with #690. To overcome this I rebased the ceremony branch onto main and then created a new ceremony branch (with -rebased suffix) and opened a PR #692

We should decide on what the resolution strategy should be and document this.

The merge conflict appeared due to the automatic snapshot and timestamp job running after the ceremony started, and before it was merged into main. It's scheduled to run every 7th day of the week, like the 28th which was the day for this signing ceremony.

I suggest to update the cron job to run 10.00 on every Monday instead: 0 10 * * 1, see: https://github.com/sigstore/root-signing/blob/main/.github/workflows/stable-snapshot-timestamp.yml#L25

[asraa]

Thank you for filing this issue!

I see two ways we can proceed - we could freeze the job during the ceremony (and ensure that there is no expiration failure - perhaps something like "freeze, unless x days out" OR add a pre-ceremony step of running the automation so that there's no change main can expire)

I've handled some repos where they freeze automation during a release - I could set something up.

Running the cron on Monday instead seems like a decent solution as well, but might still end up with issues in case a ceremony bleeds over to the next weekend, right?

[kommendorkapten]

Yes, running the cron on Monday is not what I see a solution, it's just have it a bit more deterministic when it will run. We still need to have good procedures do deal with merge conflicts.

[asraa]

How about something like this? Right now ceremony init happens through the initialize workflow, and the ceremony end is marked by the merge into main - both these workflows can trigger some state to set/unset the automated snapshot/timestamp.

I'll look into what that state could be. It could be an open issue? "Complete ceremony X on branch X" which can open and close during the ceremony (and perhaps we can use that as the ceremony coordination/notes issue).

[kommendorkapten]

Yes, I think that is a good idea. I'm thinking if we can have it simple, like if there exists a ceremony/YYYY-MM-DD branch (not test-ceremony) and YYYY-MM-DD is closer than one week don't run the snapshot and staging? So then we don't need to manually set a state, the existence of such a branch is the indication. Of course we should have the option to override this check in an emergency situation.

[asraa]

Of course we should have the option to override this check in an emergency situation.

Agree, and we can do that by ensuring that we still allow workflow_dispatch to occur & ensure that it's only blocked if the valid-until date is far out.

Okay, will try to complete this next week! I'll mark this as a v7 milestone just so we prioritize it.