copilot 1.8: variables and secrets in environments section of manifest not populated from pipeline

[sugarjig]

I just upgraded to Copilot 1.8. I have a pipeline that deploys three services. I just noticed that when the pipeline deployed the services after the upgrade, any variables or secrets defined in the environments section of the service's manifest no longer have their values populated in the running containers. In my case, these are things like the DB host and DB password, which differ between environments. The result is that the application is no longer connected to the database.

Any variables or secrets that are not defined in environments are still populated. Also, any secrets created by addon outputs still have their values populated.

When I inspect the task definition in ECS, I can confirm that it no longer contains the variables and secrets that I am expecting, but it does contain the variables not in the environments section.

When I run copilot svc deploy from my workstation, all the values are populated as expected.

[bvtujo]

Uh oh, this looks like a bug on our end. Can you run copilot svc package --output-dir svcname --env offendingenvironment from your workstation and check whether the Environment and Secrets sections of the Cloudformation generated at ./svcname/svcname-env.stack.yml are correctly populated?

In the pipeline, we use a CloudFormation deployment stage instead of copilot svc deploy, so we use copilot svc package to render the CloudFormation.

[sugarjig]

After running the package command and inspecting the .stack.yml file, I can see that under the TaskDefinition section, I don't see all the variables that I expect. It matches what I described above.

[Lou1415926]

Hi @sugarjig! This is a bug on our end. I am so sorry for the trouble it caused 🙇🏼 !

The bug is caused by this attempt to fix the old image build bug, resulting in environment overrides being only applied when you run svc deploy.

While we are working on a patch release, there are two ways to mitigate the issue for you:

1. If you don't need any feature from v1.8, could you keep using v1.7 for now?
2. Alternatively, you could create multiple manifests, one for each environment (e.g. manifest-test.yml and manifest-prod.yml). In each manifest, specify your environment configuration as the top level configuration. Then, in the pipeline code, loop over the environments, renaming each manifest-env.yml to manifest.yml so that svc package can pick up the correct environment manifest file.

Again I apologize for the churn 🙇🏼 I am working on the fix now!

[sugarjig]

I'll give 1.7 a try. Thanks for looking into it!

[efekarakus]

Hi @sugarjig, we just release v1.8.1 that should fix this issue: https://github.com/aws/copilot-cli/releases/tag/v1.8.1

[sugarjig]

@efekarakus Great! I'll give it a try. Thank you!