[ECS] [feature]: Support for versioned secrets in TaskDefinitions

[yavor-atanasov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Working with secrets in Fargate(ECS) involves writing a secret in Secrets Manager or SSM and then referring to it in your container definition, e.g. with SSM:

{
  "containerDefinitions": [{
    "secrets": [{
      "name": "environment_variable_name",
      "valueFrom": "arn:aws:ssm:region:aws_account_id:parameter/parameter_name"
    }]
  }]
}

Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html

However, we are not able to make use of the versioning capabilities of SSM in order to achieve immutable task definitions. It would be great if we are able to create a task definition with a specific version of a secret, e.g.:

{
  "containerDefinitions": [{
    "secrets": [{
      "name": "environment_variable_name",
      "valueFrom": "arn:aws:ssm:region:aws_account_id:parameter/parameter_name:version"
    }]
  }]
}

This will help with creating immutable deployments with Fargate and be able to reliably rollback just by just switching the TaskDefinition version of a Service.

NOTE: I realise this request might depend on whether or not one is able to reference secret versions via an ARN with SSM (or SecretsManager). It is unclear from their documentation and I can't really see any examples of using ARNs with versions for those services.

Which service(s) is this request for?
Fargate (ECS)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We are trying to have a simple and reliable mechanism to do immutable deployments and rollbacks with Fargate. It looks like TaskDefinitions are the right level of abstraction for this and the fact that they can be versioned hints at that. However in order to make TaskDefinition versions useful we need 3 things inside them to be immutable:

• the container images used in the ContainerDefinitions
• non-secret configuration in terms of plaintext environment variables
• secrets

Non-secret environment variables are not an issue as they get "baked" into the task definition. Container images are a little bit of an issue because if you are using a mutable tag URL for an image (e.g. :latest) then the TaskDefinition version is created with that (as opposed to resolving to the immutable @sha256 url) which means your Service might end up with heterogeneous tasks even though they were created from the same TaskDefinition. This we can work around by enforcing the image URLs we use in task definitions to always be the @sha256 ones.

Secrets remain a problem. Since we are using CloudFormation for all this, we tried using "dynamic references" and inject SSM secrets into the normal Environment variables:

 "environment": [{
      "name": "secret_foo",
      "value": "{{resolve:ssm-secure:foo:1}}"
    }]

However this errors with: SSM Secure reference is not supported in: [AWS::ECS::TaskDefinition/Properties/ContainerDefinitions]
Note that even if this worked, it would mean a slightly different permissions model regarding the KMS key used to encrypt those secrets. Ideally we'd like to allow only the task execution role to be able to decrypt those secrets, as opposed to expand that to a role used by CloudFormation when updating the stack.

Are you currently working around this issue?
We know the following work around, however we don't really like it: Push code into each image that runs as an entry point and pulls all required secrets via the aws cli. This will allow us to target specific SSM parameter versions. However this means we won't be using the standard Secrets mechanism for TaskDefinitions, it increases the number of moving parts and also we are software engineers that hate writing code as any increase in code surface area introduces potential for problems.

Additional context
For what it's worth, we are using CloudFormation for all this - we don't like clicking buttons on the UI or interacting imperatively with the AWS APIs directly. We like to declaratively define our infrastructure, version it and let CloudFormation deal with the underlying APIs.

[srrengar]

This has been launched for tasks using the EC2 launch type. We are currently working on the capability for tasks using Fargate.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html#secrets-envvar
^See the example on 'referencing a specific secret version'

[yavor-atanasov]

Thank you for the update. I looked at the doc and seems to fit the requirement. Interesting that the launch type (EC2 vs Fargate) matters in the implementation. I would've thought this is just a change at the AWS::ECS::TaskDefinition level and any abstractions above that (like what runs those tasks) don't matter.

Just to clarify - has AWS Parameter Store remained out of scope for this? If yes, could you give some detail on the reasons behind it.

[srrengar]

Hi folks, this has now launched. Thanks for your patience
https://aws.amazon.com/about-aws/whats-new/2020/11/aws-fargate-for-amazon-ecs-launches-features-focused-on-configuration-and-metrics/

[Nevon]

I would like for this to be re-opened since this has only been implemented for Secrets Manager, whereas the original issue referenced Parameter Store. Currently there is no way to use a versioned secret from parameter store in a task definition, and the alternative is to migrate to secrets manager at 8x the cost.

[ei-grad]

Feels like the ECS is now like a legacy product lacking progress a in community-anticipated minor issues :-(.