Security and Maintenance

Service Account Key Rotation

Every three months, for security purposes, we must rotate the keys for the deployer service account.

• the deployer SA is used by Jenkins during a release to deploy Import Service to App Engine.

The steps below describe the process for rotating keys.

You should do these steps anywhere between 75 and 90 days since the key was last rotated. Before 75 days, Terraform will not rotate the key.

Rotate via Terraform

Create a new PR in https://github.com/broadinstitute/terraform-ap-deployments targeting import service. You can do this by:

• changing something trivial and inconsequential in the import-service/ Terraform code, such as a description, or
• explicitly telling atlantis plan which Terraform project to look at, using the -p flag (see https://www.runatlantis.io/docs/using-atlantis.html#options)

Run atlantis plan in your PR against the import-service projects. The plan output should show that Terraform will rotate the key for the deployer SA.

Run atlantis apply in your PR against the import-service projects. This will perform the key rotation and save the new key to Vault.

Smoke test

• To verify the new deployer key, manually trigger a deployment of Import Service to production by running the import-service-manual-deploy job in prod Jenkins. If the deployment succeeds, the deployer key rotation worked.
  • Clicking "Rebuild" on the last-successful Jenkins deploy job will re-deploy the same code that is currently in production; this is a simple route to smoke-testing deploys.
  • Since this deployed a new GAE version of Import Service, also smoke test Terra prod by uploading a large/async TSV and seeing it succeed.

Prepare for the next rotation

• Set a Slack reminder on #dsp-analysis-journeys channel for the next rotation by typing something like /remind :alert_red_light: Reminder to rotate the service account key for Import Service by following these (https://github.com/broadinstitute/import-service/wiki/Security-and-Maintenance) instructions in 85 days.