Dogfood upcoming release during code freeze

[spikecurtis]

Proposal

We could dramatically increase the amount of time human users spend with the to-be-released code by running it on dogfood between code freeze and release day.

• This creates a forcing function to ensure that bugs are quickly fixed and cherry-picked
• It gives product and marketing a stable place to see the “final draft” of features in the release
• It will give us much higher confidence that big, embarrassing, “ship stopper” bugs are not present.
• Less frequent deploys will make it easier to spot resource leaks or other problems of attrition

Implementation Details

Deploy scripts contain logic to switch from main to the latest release branch during the freeze.

1. Algorithm:
   1. list branches and filter for release/(\d+\.\d+) , then sort by major/minor to get the latest release.
   2. If tag v<MAJOR>.<MINOR>.0 exists, we deploy from main. If not we deploy from release/<MAJOR>.<MINOR>
2. Embodiment
   1. Above algorithm is implemented as a bash script in our scripts directory.
   2. A new job, should-deploy in the ci.yaml GitHub actions calls the script and determines whether we should deploy
   3. The deploy job is predicated on the output of should-deploy

Special care needs to be taken with container registry image tags. Dogfood is always deployed from ghcr.io/coder/coder-preview:latest.

• latest should ALWAYS track the main branch, e.g. we still build and push the latest tag on main always even if we're not deploying it
  • if another branch is being deployed instead, it should NEVER be pushed to to the latest tag
• a new tag dogfood should always track the latest build in whatever branch we're using for dogfood, either main or a release/ branch
• Dogfood should be updated to use the dogfood tag in the values file

Relates to #1024

[deansheather]

Dogfood currently deploys directly from ghcr.io/coder/coder-preview:latest since the deployment needs to be automated without committing to the dogfood repo: https://github.com/coder/dogfood/blob/f5b3155340e7dc432b49726990eb8ce3179cc7ad/clusters/dogfood-v2/coder/server/configs/values.yaml#L14-L17

For now we can switch this tag to be named dogfood and retag whatever we just built during deploy or something. A better long term solution can be done with the image automation stuff built into Flux during the "centralized deployment" phase of the dogfood glow up RFC

[deansheather]

BTW I don't think I have time to do this for this release freeze, and this is not something I'd want to rush anyway.

edit: should be fine now I've split it

[deansheather]

I split this issue up since automated cutting is much more difficult and also not necessary for the primary goal