Skip to content

Concourse Upgrade

Jump to bottom
Wesley Hindle edited this page Mar 12, 2026 · 5 revisions

This page will mention considerations for when Concourse is upgraded. In most cases you can ignore the content here as Concourse updates - even major ones - don't tend to be breaking. This page assumes you are familiar with notifications-concourse, notifications-concourse-deployment.

Previous upgrade ticket contains more details if required.

General consideration

Killing off instances manually

After the scale-out job you may get issues with concourse-web e.g. UI not working. This is because the old instances have not yet scaled in and there is some compatibility issue between workers using the old version and the new version. In some cases the scale-in action will be triggered. You can check this in the relevant autoscaling groups tab in the AWS console. It is also safe to manually terminate the old workers.

Re triggering whole pipelines.

Sometimes after upgrading the input requirements for a job may never be satisfied, desipite looking healthy. You should start the pipeline again from the beginning.

Rolling all workers

You must roll all workers after upgrading the concourse version. Otherwise they will roll after hours and any issues may cause an on-call alert.

Dev env rebasing against main

In-use dev envs may require a rebase against notifications-aws if the pipeline config has changed as part of the new Concourse version.

Testing the deployment bag

Our main pipeline does not use the concourse-bag-resource, but it must be tested for compatibility before we roll out any upgrades. This is a custom resource written by Rob.

These PRs:

  1. https://github.com/alphagov/notifications-concourse-deployment/pull/117/changes
  2. https://github.com/alphagov/notifications-concourse/pull/45/changes 3.https://github.com/alphagov/notifications-concourse/pull/44/changes

contains the config required to test the deployment bag in concourse-staging. There are also some manual changes which need to happen. This setup will test the get actions of this resource, but not the put ones. Historically, if the get actions pass it has been proof enough that the deployment bag resource is compatible. Getting the put actions working is not covered here, but would be a more involved process, giving the Concourse team the relevant key.

Pull deployment bag image

  1. in notify-deploy-staging manually create ECR repo, concourse-bag-resource.

  2. Log into ECR in notify-deploy

gds aws notify-deploy -- aws ecr get-login-password --region eu-west-1 \
| docker login \
--username AWS \
--password-stdin 975210404583.dkr.ecr.eu-west-1.amazonaws.com

  1. Pull the image down docker pull .dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest`

  2. Log into notify-deploy-staging's ECR

gds aws notify-deploy-staging-admin -- aws ecr get-login-password --region eu-west-1 \
| docker login \
--username AWS \
--password-stdin 390844751771.dkr.ecr.eu-west-1.amazonaws.com
  1. Retag the old image in the format docker tag <aws account id you pulled from >.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest <aws account if you want to push to>.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest e.g.
docker tag 975210404583.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest 390844751771.dkr.ecr.eu-west-1.amazonaws.com/concourse-bag-resource:latest
  1. Push the image to the notify-deploy-staging account
gds aws notify-deploy-staging-admin -- docker push 390844751771.dkr.ecr.eu-west-1.amazonaws.com/notifications-api:2026-03-05-11.29.40-c405671ce8d3ccfbcb59647810cb5f07215a3088
  1. Check your concourse-deployment-bag image exists in the eu-west-1 region.

Correct permissions

  1. The main team is accessed with an AWS user concourse-<staging>-main-concourse-worker-static user. This user must have access to the concourse-bag-resource a policy like this will do
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "ecr:GetAuthorizationToken",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ]
}
  1. Check the permissions on the concourse-bag-repo allow the concourse-<staging>-main-concourse-worker-static principal to access the required resources e.g.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Access_deployment_bag",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::390844751771:user/concourse-staging-main-concourse-worker-static"
      },
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeImageScanFindings",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:ListTagsForResource",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ]
    }
  ]
}

Databse rollbacks

The Concourse DB can be rolled out via the same Terraform mechanism we have in notifications-aws. Instructions for how to do that can be found here.

If you decide to create this database and switch over to it locally you must commit this change and update the concourse-staging branches for notifications-concourse and notifications-concourse-deployment or when you next run the pipeline you will destroy the database. If you're paranoid you can pause the deploy job.

If your Concourse upgrade contains schema changes then you must take a snapshot of the DB using the old schema beforehand. You should clean these up at some point after the upgrade is deployed.

Clone this wiki locally