Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ECS][Fargate]: Add EFS volumes through CloudFormation #825

Closed
srrengar opened this issue Apr 8, 2020 · 41 comments
Closed

[ECS][Fargate]: Add EFS volumes through CloudFormation #825

srrengar opened this issue Apr 8, 2020 · 41 comments
Assignees
Labels
ECS Amazon Elastic Container Service Fargate AWS Fargate

Comments

@srrengar
Copy link

srrengar commented Apr 8, 2020

Community Note

  • Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
CloudFormation support for adding EFS volumes to ECS tasks on Fargate or EC2.

Which service(s) is this request for?
Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
EFS support launched for ECS on Fargate and EC2. Customers deploying through CloudFormation should also be able to use this feature.

@srrengar srrengar added the Proposed Community submitted issue label Apr 8, 2020
@srrengar srrengar changed the title [ECS][Fargate][CFN support for EFS]: Add EFS volumes through CloudFormation [ECS][Fargate]: Add EFS volumes through CloudFormation Apr 8, 2020
@srrengar srrengar added ECS Amazon Elastic Container Service Fargate AWS Fargate Work in Progress and removed Proposed Community submitted issue labels Apr 8, 2020
@rpgreen
Copy link

rpgreen commented Apr 16, 2020

Is there a timeline on this? It is disappointing to see feature releases without CFN support on day 1. We will have to jump through some hoops to make EFS work with our current setup.

@gary-cowell
Copy link

And I need it to work through the CDK. Which requires cloudformation first.

@srrengar srrengar self-assigned this Apr 20, 2020
@guillaumesmo
Copy link

Please note you can use custom resources while a feature is not yet supported by CloudFormation

I wrote a simple proof of concept to create an ECS task definition with EFS mount through CloudFormation:
https://gist.github.com/guillaumesmo/4782e26500a3ac768888daab3c55b139

@Trandel
Copy link

Trandel commented May 7, 2020

馃憤

@jedis00
Copy link

jedis00 commented May 12, 2020

Any update on this one?

@machielg
Copy link

I'm using a custom resource but the sad thing is that I need to replicate the whole task definition creation in stead of just amending the volume part. This issue is also blocking the inclusion into CDK. aws/aws-cdk#6918

@skysb
Copy link

skysb commented May 20, 2020

Hi guys, is there an update on when the cloudformation would be updated with the EFS inclusion for ECS?

@srrengar
Copy link
Author

Hi everyone, thank you for the valuable feedback. I can assure you that this is coming soon and we will provide updates here when we have more to share. We decided to release EFS for ECS (on both EC2 and Fargate) sooner so that customers can use it and provide early feedback instead of waiting till CloudFormation was ready. We understand CloudFormation is a great way to create and update resources to use new features, and we are working on improving how we release CloudFormation support for new features going forward.

@cajames
Copy link

cajames commented May 30, 2020

if anyone's still looking for a way to do this in CDK, here's a sample gist with a workaround, going down the path @guillaumesmo mentions above: https://gist.github.com/cajames/3daec680b1101c8358e2ff30dfadd52a

The key takeaway, creating a custom resource Task Definition, and then using CDK "Raw Overrides" to attach it to the service. Documented here: https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html

I've explained the method here: aws/aws-cdk#6240 (comment)

Hope this can help anyone. This is a temporary workaround, and once this issue is resolved, it'll be an easier switch to remove the custom task definition and add the config directly into the service. :)

@bellaxievmlyr
Copy link

Any update on how long it would take? A few days or weeks or months? I'm currently blocked by this issue. If it's a few days, I might just wait rather than spending a few days to figure out a workaround.

@pavneeta pavneeta added this to Researching in containers-roadmap via automation Jun 25, 2020
@pavneeta pavneeta moved this from Researching to Coming Soon in containers-roadmap Jun 25, 2020
@machielg
Copy link

machielg commented Jul 1, 2020

Any update on how long it would take? A few days or weeks or months? I'm currently blocked by this issue. If it's a few days, I might just wait rather than spending a few days to figure out a workaround.

It moved from 鈥榠n progress鈥 to 鈥榗oming soon鈥 which also includes issues from 2019. Not a good sign 馃

@rpgreen
Copy link

rpgreen commented Jul 15, 2020

Folks, it's been over 3 months since this feature was launched and there's STILL no CFN support. This will definitely make me think twice about adopting new ECS features in the future. CFN support is not a 'nice to have', it should be included on launch.

@Samseppiol
Copy link

Folks, it's been over 3 months since this feature was launched and there's STILL no CFN support. This will definitely make me think twice about adopting new ECS features in the future. CFN support is not a 'nice to have', it should be included on launch.

Are we thinking about the same company? This is AWS, cfn support isn't for atleast 12-18 months after announcing clickops.

@sandeepboyapati
Copy link

eagerly waiting

@bushong1
Copy link

It's 2020. CloudFormation has been out for 9 years. How is this still happening?

@ghferrari
Copy link

I'm eagerly looking forward to this, too and I hope we'll be kept regularly updated on progress.

Just to give a bit of feedback to @srrengar , in my view there is simply no such thing as "releasing a feature" without cloudformation support. As far as I'm concerned, either it's in cloudformation where it can be documented, placed in version control and reliably, repeatedly tested and deployed or it doesn't exist. As a sole developer, I don't have time for manual features and wouldn't even consider using them.

So this is me giving some feedback and a big show of encouragement to the idea of "reducing the time between a feature being released and getting cloudformation support". But I'd really like you to go all the way and understand that for many of us, a new feature is only really released when it becomes available in cloudformation.

@craighurley
Copy link

All these small straws triggered my move to terraform. So far so good, plus I get to skill up on a tool that can deploy to other cloud providers.

@gary-cowell
Copy link

Just to give a bit of feedback to @srrengar , in my view there is simply no such thing as "releasing a feature" without cloudformation support. As far as I'm concerned, either it's in cloudformation where it can be documented, placed in version control and reliably, repeatedly tested and deployed or it doesn't exist. As a sole developer, I don't have time for manual features and wouldn't even consider using them.

So this is me giving some feedback and a big show of encouragement to the idea of "reducing the time between a feature being released and getting cloudformation support". But I'd really like you to go all the way and understand that for many of us, a new feature is only really released when it becomes available in cloudformation.

Even worse, is, I have the same goals of documented code in version control and repeatable reliable testing, but , I need this from the AWS CDK. As CDK relies on cloudformation, I have no doubt that any CDK release of this will be well behind any eventual cloudformation update.

[ I know you can mess with the cloudformation in CDK even without full support - I'd rather not do that ]

@mwarkentin
Copy link

Looks like this is supported now: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-volumes

@GeoffWilliams
Copy link

it works! I just hit the outdated lambda SDK issue where it rejects valid task definitions that use access points so now I can use this 馃嵕

@frank69m
Copy link

frank69m commented Aug 1, 2020

So are you guys saying that with CDK 1.56, I can now mount EFS volumes in my task definition.... I glanced at the docs, not many good examples

@frank69m
Copy link

frank69m commented Aug 1, 2020

Ok. So I just looked at the docs. I'm looking at mounting an EFS volume to a Fargate task definition with the cdk

It always seems to revert to Bind Mount

in my TD for python

volume=[
{
"name": "EFS",
"efsVolumeConfiguration": {
"fileSystemId": mount.file_system_id,
"rootDirectory": "/"
}
}
]

It does a synth fine, but when I checkout the TD, it creates the name, but it's still stuck at Bind Mount

any ideas? Please help

@srrengar
Copy link
Author

srrengar commented Aug 2, 2020

hey everyone, this is supported now. The official announcement will be published Monday

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-efsvolumeconfiguration.html

@ryanchapman
Copy link

ryanchapman commented Aug 2, 2020

Since it took me a few hours to glue CFN resources together to get this working, thought I'd post it here. Hopefully it saves others some time.

If you run into unsuccessful EFS utils command execution; code: 32, then you may have security group issues. Check the ingress on your sg associated with your mount targets (GrafanaEFSServerSecurityGroup in this example) as well as egress on the sg associated with your ECS service (GrafanaTaskSecurityGroup in this example).

AWSTemplateFormatVersion: 2010-09-09
[...]
Resources:
[...]
  GrafanaEFSVolume:
    Type: AWS::EFS::FileSystem
    Properties:
      Encrypted: true
      FileSystemTags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-efs-vol"

  GrafanaEFSServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${AWS::StackName}-efs-server-endpoints"
      GroupDescription: Which client ip addrs are allowed to access EFS server endpoints for grafana mount
      VpcId:
        Fn::ImportValue: !Sub "${VPCStack}-VPC"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp:
            Fn::ImportValue: !Sub "${VPCStack}-CIDR"
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-efs-server"

  GrafanaEFSMountTarget0:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 0
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaEFSMountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 1
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaEFSMountTarget2:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 2
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn:
      - ECSExecutionPolicy
    Properties:
      Volumes:
        - Name: grafana
          EFSVolumeConfiguration:
            FilesystemId: !Ref GrafanaEFSVolume
            TransitEncryption: ENABLED
      ContainerDefinitions:
        - Name: grafana
          Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${GrafanaImage}"
          # Running as root to set perms on EFS mount before running grafana
          User: root
          Entrypoint: ['bash', '-c']
          Command:
            - |
              chown -R grafana:grafana /var/lib/grafana
              su grafana -s /bin/bash -c /run.sh
          MountPoints:
            - SourceVolume: grafana
              ContainerPath: '/var/lib/grafana'
              ReadOnly: false
          StopTimeout: 60
          PortMappings:
            - ContainerPort: 3000
              HostPort: 3000
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref 'AWS::Region'
              awslogs-group: !Ref ECSLogGroup
              awslogs-stream-prefix: ecs-grafana
      Cpu: '512'
      Memory: '2048'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !GetAtt ECSExecutionRole.Arn
      TaskRoleArn: !GetAtt ECSTaskRole.Arn

  GrafanaTaskSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${AWS::StackName}-grafana-task"
      GroupDescription: Security group for the grafana task.
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 3000
          ToPort: 3000
          SourceSecurityGroupId:
            Fn::ImportValue: !Sub "${PlatformStack}-InternalLoadBalancerServerSecurityGroup"
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp:
            Fn::ImportValue: !Sub "${VPCStack}-CIDR"
      VpcId:
        Fn::ImportValue: !Sub "${VPCStack}-VPC"

  GrafanaService:
    Type: AWS::ECS::Service
    Properties:
      Cluster:
        Fn::ImportValue: !Sub "${PlatformStack}-ECSCluster"
      DesiredCount: 1
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      LaunchType: FARGATE
      LoadBalancers:
        - ContainerName: grafana
          ContainerPort: 3000
          TargetGroupArn: !Ref InternalLoadBalancerTargetGroup
      NetworkConfiguration:
        AwsvpcConfiguration:
          SecurityGroups:
            - Fn::ImportValue: !Sub "${PlatformStack}-InternalLoadBalancerClientSecurityGroup"
            - !Ref GrafanaTaskSecurityGroup
          Subnets:
            Fn::Split:
              - ','
              - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"
      PlatformVersion: '1.4.0'
      TaskDefinition: !Ref GrafanaTaskDefinition

@frank69m
Copy link

frank69m commented Aug 2, 2020

Has anyone confirmed if this works with the CDK.

Everytime I synth and try to look at my CF, it only shows

"Volumes": [
{
"Name": "testefs"
}
]

And yes, I did put in the efsVolumeConfiguration

@engineal
Copy link

engineal commented Aug 3, 2020

Has anyone confirmed if this works with the CDK.

Everytime I synth and try to look at my CF, it only shows

"Volumes": [
{
"Name": "testefs"
}
]

And yes, I did put in the efsVolumeConfiguration

@frank69m While I don't think this feature has been added to the CDK generated CFN resources or higher level constructs yet, I was able to use this feature through the CDK escape hatch: https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html

To my Fargate task definition I added a property override:

const cfnTask = this.task.node.defaultChild as CfnTaskDefinition;
cfnTask.addPropertyOverride('Volumes', [{
    EFSVolumeConfiguration: {
        FilesystemId: this.fileSystem.fileSystemId,
        TransitEncryption: 'ENABLED'
    },
    Name: 'efs'
}]);

and to the container added to that task definition I added a mount point:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

I have successfully used this to add an EFS mount to a Fargate task.

@cmoreno82
Copy link

I still have only the "docker_volume_configuration" option with CDK 1.56.0 and python. I use the ecs.FargateTaskDefinition.add_volume function:

Task Definition

      fargate_task_definition_zk = ecs.FargateTaskDefinition(
          self, "task_" + zk_name,
          family="task-" + zk_name,
          execution_role=self.role,
          memory_limit_mib=self.node.try_get_context("zookeeper_memory"),
          cpu=self.node.try_get_context("zookeeper_cpu")
      )     
       fargate_task_definition_zk.add_volume(
          name="test",
          EFSVolume....  <--- Only "docker_volume_configuration" is possible
      )

Someone else test it with cdk+python?

@srrengar
Copy link
Author

srrengar commented Aug 3, 2020

Here is the official announcement - https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-ecs-announces-cloudformation-support-for-amazon-efs-volumes/

@srrengar srrengar closed this as completed Aug 3, 2020
containers-roadmap automation moved this from Coming Soon to Just Shipped Aug 3, 2020
@frank69m
Copy link

frank69m commented Aug 3, 2020

great. works now with CDK and the @engineal solution above.

@paveljos
Copy link

paveljos commented Aug 5, 2020

Just to close the gap for anyone looking for a python solution based on @engineal 's submission, here's what I implemented in CDK:

cfn_task = self.task_definition.node.default_child
cfn_task.add_property_override("Volumes", [{
   "EFSVolumeConfiguration": {
      "FilesystemId": efs_id.file_system_id,
      "TransitEncryption": "ENABLED"
   },
   "Name": "efs"
}])

@skindc
Copy link

skindc commented Nov 29, 2020

For anyone still having issues with this, I seems ensuring that you explicitly set the platform version in your containers definition to '1.4.0' solves the problem. When using 'latest' this is actually implying 1.3.0 which seems does not support the latest announcements.

@yogeshjoshi
Copy link

For anyone still having issues with this, I seems ensuring that you explicitly set the platform version in your containers definition to '1.4.0' solves the problem. When using 'latest' this is actually implying 1.3.0 which seems does not support the latest announcements.

Yes @skindc , yes this has been solved in version 1.4.0 in August this year.
Latest version in definition refers 1.3.0

@mreferre
Copy link

@skindc / @yogeshjoshi tangential to this thread, we have an open issue around how to better manage/communicate new Platform Versions. If you have opinions or feedback please leave a comment at #1069 . Thanks!

@eob
Copy link

eob commented Jul 16, 2021

@engineal I'm using your solution above, but when using ecs_patterns.ApplicationLoadBalancedFargateService with the taskImageOptions I can't figure out how to access my container object, as in:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

Do you know if it's still possible to attach an EFS volume to a container that has been created with:

      taskImageOptions: { 
        image: ContainerImage.fromEcrRepository(...),
        environment: { ... },
        containerPort: 8080,
        enableLogging: true,
      },

@engineal
Copy link

engineal commented Jul 16, 2021

Hi @eob and anyone else who is interested. I believe the property override I had suggested before isn't required anymore, as these props are now supported by the aws-cdk constructs.

I can replace my previous use of the property override with a call to the addVolume method on my task definition:

this.task.addVolume({
    name: "efs",
    efsVolumeConfiguration: {
        fileSystemId: this.fileSystem.fileSystemId,
        transitEncryption: "ENABLED"
    }
});

And just like before, to the container added to that task definition I added a mount point referencing that volume:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

Now for your example @eob, if you've created your service using the ecs-patterns ApplicationLoadBalancedFargateService construct like so:

const service = new ApplicationLoadBalancedFargateService(this, 'service', {
    taskImageOptions: {
        image: ContainerImage.fromEcrRepository(...),
        environment: { ... },
        containerPort: 8080,
        enableLogging: true,
    }
});

you can access the task definition created by that service using service.taskDefinition, and add the EFS volume to the task definition like so:

service.taskDefinition.addVolume({
    name: "efs",
    efsVolumeConfiguration: {
        fileSystemId: this.fileSystem.fileSystemId,
        transitEncryption: "ENABLED"
    }
});

Now the ApplicationLoadBalancedFargateService construct adds a single container to the task definition, which will either be named 'web' or the value you assign containerName in your taskImageOptions. This means you can access that container by using taskDefinition.findContainer('web') (or taskDefinition.findContainer(containerName) if you specified containerName in your taskImageOptions). Additionally, since the container added by the ApplicationLoadBalancedFargateService is the first container added, you can access it by using taskDefinition.defaultContainer as well.

So, to add the mount point to the container created by the ApplicationLoadBalancedFargateService, you can do:

service.taskDefinition.findContainer('web')?.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

or

service.taskDefinition.defaultContainer?.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

@eob
Copy link

eob commented Jul 21, 2021

@engineal thank you for so kindly explaining that!

Your comment got everything deploying properly, but I'm still getting permissions errors when I try to actually access the volume.

I've seen a few articles about configuring AccessPoints with EFS, but most Fargate tutorials proceed without them as if they're not actually necessary. Do you know if there any permissions gotchas that I might not be noticing?

This is what my code looks like -- I copy-paste-customized the ApplicationLoadBalancedFargateService code so that it's easier to just pass in mounts as an option.

// FILESYSTEM
const fs = new efs.FileSystem(this, "IndexFilesystem", {
  vpc: vpc,
  performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,
  throughputMode: efs.ThroughputMode.BURSTING,
  fileSystemName: "indexFilesystem"
})
  
// VOLUME
const volume = {      
  name: "indexVolume",
  efsVolumeConfiguration: {
    fileSystemId: fs.fileSystemId,
    TransitEncryption: 'ENABLED',
  },
};

// this is a FargateTaskDefinition object
this.taskDefinition.addVolume(volume)  


// MOUNT POINT
const mountPoint = {
  containerPath: "/data",
  sourceVolume: volume.name,
  readOnly: false,
};

// This is the container created inside ApplicationLoadBalancedFargateService
container.addMountPoints(mountPoint)

// PROVIDE ACCESS
// this.service is the FargateService
this.service.connections.allowFrom(fs, Port.tcp(2049));
this.service.connections.allowTo(fs, Port.tcp(2049));

@engineal
Copy link

@engineal thank you for so kindly explaining that!

Your comment got everything deploying properly, but I'm still getting permissions errors when I try to actually access the volume.

I've seen a few articles about configuring AccessPoints with EFS, but most Fargate tutorials proceed without them as if they're not actually necessary. Do you know if there any permissions gotchas that I might not be noticing?

This is what my code looks like -- I copy-paste-customized the ApplicationLoadBalancedFargateService code so that it's easier to just pass in mounts as an option.

@eob Would you happen to be able to describe or include in a comment the error message that you receive?

@sukrit007
Copy link

@eob What is the user (UID/GID) of the user running within the container? You might have to create an access point for specific Posix user to get around Posix permissions. https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html

@fire015
Copy link

fire015 commented Oct 28, 2021

Just adding to what @eob and @sukrit007 said, if you want to use an access point in your Fargate service so that you can mount as non-root here is what I did (assume 2000 is the POSIX UID that runs inside the container):

const fs = new efs.FileSystem(this, "Filesystem", {
  vpc: vpc,
  performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,
  throughputMode: efs.ThroughputMode.BURSTING,
  fileSystemName: "MyFilesystem",
});

const accessPoint = new efs.AccessPoint(this, "AccessPoint", {
  fileSystem: fs,
  path: "/data",
  createAcl: {
    ownerUid: "2000",
    ownerGid: "2000",
    permissions: "0755",
  },
  posixUser: {
    uid: "2000",
    gid: "2000",
  },
});

const volume: ecs.Volume = {
  name: "MyVolume",
  efsVolumeConfiguration: {
    fileSystemId: fs.fileSystemId,
    transitEncryption: "ENABLED",
    authorizationConfig: {
      accessPointId: accessPoint.accessPointId,
    },
  },
};

fargateService.taskDefinition.addVolume(volume);

fargateService.taskDefinition.defaultContainer!.addMountPoints({
  containerPath: "/data",
  sourceVolume: volume.name,
  readOnly: false,
});

fargateService.service.connections.allowFrom(fs, ec2.Port.tcp(efs.FileSystem.DEFAULT_PORT));
fargateService.service.connections.allowTo(fs, ec2.Port.tcp(efs.FileSystem.DEFAULT_PORT));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ECS Amazon Elastic Container Service Fargate AWS Fargate
Projects
containers-roadmap
  
Just Shipped
Development

No branches or pull requests