Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Publisher deployment #152

Merged
merged 5 commits into from
Feb 3, 2021
Merged

Refactor Publisher deployment #152

merged 5 commits into from
Feb 3, 2021

Conversation

bilbof
Copy link
Contributor

@bilbof bilbof commented Feb 3, 2021

This rewrites the way we deploy the Publisher app, following on from the refactor of content store in PR #150.

The intent is to simplify the way we deploy publisher (web and worker) and run tasks using the publisher task definition.

This change:

  • removes the deployment modules for publisher-web and publisher-worker
  • adds logic to the govuk-publishing-platform/publisher_app file to generate the JSON required to register the web and worker task definitions for publisher
  • removes the now unnecessary task definition module for publisher
  • modifies the deploy pipeline for publisher to register the publisher task definitions using the AWS CLI, rather than Terraform
  • modifies the deploy pipeline to store the JSON generated by the big Terraform apply (govuk-publishing-platform) in S3, so we can use it to register task definitions, overriding only the IMAGE key.
  • modifies the deploy pipeline so that we can run db migrations for publisher, which will break smokey, and anything that uses run-task temporarily.

This introduces additional complexity into the Concourse pipeline (particularly run-task is not that simple), I'll address this in a future PR.

@bilbof bilbof force-pushed the bilbof/publisher-refactor branch from 78768d5 to e4beab2 Compare February 3, 2021 14:21
Copy link
Contributor

@richardTowers richardTowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 looks absolutely amazing, thanks!

I spotted a few minor things, but generally this looks great.

concourse/pipelines/deploy.yml Outdated Show resolved Hide resolved
terraform/deployments/apps/variables.tf Outdated Show resolved Hide resolved
terraform/deployments/govuk-publishing-platform/outputs.tf Outdated Show resolved Hide resolved
terraform/modules/monitoring/grafana.tf Outdated Show resolved Hide resolved
@bilbof bilbof force-pushed the bilbof/publisher-refactor branch 2 times, most recently from be1338a to a43d52d Compare February 3, 2021 15:42
Bill Franklin added 3 commits February 3, 2021 15:43
Makes the domain consistently named throughout the codebase.
We had a suprising number of variable names for just three domains.

New names:

```
external_app_domain = test.govuk.digital
internal_app_domain = test.govuk-internal.digital
publishing_service_domain = test.publishing.service.gov.uk
```
Container names don't need to include the app running - they just need to
match the container name specified on any load balancer.

This changes the name of all containers that currently use the service_name
as the container name, to 'app'. Now most ECS Services will have two containers:
envoy and app. This won't make it harder to see what container is running, since
the image is usually displayed next to the container name.
This will enable us to output container definitions with the image CMD
overridden (e.g. foreman run web / foreman run worker).
@bilbof bilbof force-pushed the bilbof/publisher-refactor branch from a43d52d to de77ada Compare February 3, 2021 15:52
@bilbof bilbof force-pushed the bilbof/publisher-refactor branch from de77ada to 80f32ae Compare February 3, 2021 16:05
Bill Franklin added 2 commits February 3, 2021 16:10
This rewrites the way we deploy the Publisher app, following on from the
refactor of content store in PR #150.

The intent is to simplify the way we deploy publisher (web and worker)
and run tasks using the publisher task definition.

This change:
- removes the deployment modules for publisher-web and publisher-worker
- adds logic to the govuk-publishing-platform/publisher_app file to
  generate the JSON required to register the web and worker task definitions
  for publisher
- removes the now unecessary task definition module for publisher
- modifies the deploy pipeline for publisher to register the publisher task
  definitions using the AWS CLi, rather than Terraform
- modifies the deploy pipeline to store the JSON generated by the big
  Terraform apply (govuk-publishing-platform) in S3, so we can use it to
  register task definitions, overriding only the IMAGE key.
- modifies the deploy pipeline so that we can run db migrations for publisher,
  which will break smokey, and anything that uses run-task temporarily.

This introduces additional complexity into the Concourse pipeline,
which will be addressed in future PRs.
This changes the Concourse pipeline so that we can deploy Publisher
in the new way - rather than having a Terraform apply create the
task definitions for Publisher, we're using the AWS CLI for this bit).

The Concourse pipeline is a bit more complex and verbose than it
would ideally be. The smoke tests, and any other task using run-task
is also broken by this commit, but those should soon be fixed.
@bilbof bilbof force-pushed the bilbof/publisher-refactor branch from 80f32ae to deeef9d Compare February 3, 2021 16:10
Copy link
Contributor

@richardTowers richardTowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@bilbof bilbof merged commit 75502b0 into main Feb 3, 2021
@bilbof bilbof deleted the bilbof/publisher-refactor branch February 3, 2021 16:31
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the application task definition json when applying the govuk platforming terraform.

This PR refactors  (draft-)static app to use this new deployment style

Ref:
1. [trello card](https://trello.com/c/5JT7LE7k/382-update-the-static-application-to-use-the-new-deployment-approach)
This should look similar to #152.
We no longer register new task definitions in Terraform. Instead, our Big Terraform (:muscle:) apply outputs a task definition JSON which is passed to aws ecs register-task-definition. This should make the Terraform simpler. See PRs #150 and #152, and the design doc (internal) for the motivation behind this refactoring.
<https://github.com/alphagov/govuk-infrastructure|alphagov/govuk-infrastructure>alphagov/govuk-infrastructure | Today at 4:54 PM | Added by GitHub
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)static app to use this new deployment style.

Similar PRs are #152 and #153 for other apps.

Ref:
1. [trello
card](https://trello.com/c/5JT7LE7k/382-update-the-static-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)static app to use this new deployment style.

Similar PRs are #152 and #153 for other apps.

Ref:
1. [trello
card](https://trello.com/c/5JT7LE7k/382-update-the-static-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)static app to use this new deployment style.

Similar PRs are #152 and #153 for other apps.

Ref:
1. [trello
card](https://trello.com/c/5JT7LE7k/382-update-the-static-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 4, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 5, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 5, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/1lZxLN8V/380-update-the-router-application-to-use-the-new-deployment-approach)
fredericfran-gds added a commit that referenced this pull request Feb 5, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/1lZxLN8V/380-update-the-router-application-to-use-the-new-deployment-approach)
rtrinque pushed a commit that referenced this pull request Mar 11, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router-api app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/HDvhDo1t/379-update-the-router-api-application-to-use-the-new-deployment-approach)
rtrinque pushed a commit that referenced this pull request Mar 11, 2021
In PR #150, we refactored the way we deploy applications by outputting the
application task definition json when applying the govuk platforming terraform.
This json file is then diffed with previous version and if changes exist, a new
task definition is created for the app using AWS cli.

This PR refactors  (draft-)router app to use this new deployment style.

Similar PRs are #152, #153 and #154 for other apps.

Ref:
1. [trello card](https://trello.com/c/1lZxLN8V/380-update-the-router-application-to-use-the-new-deployment-approach)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants