Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fargate] [Volumes]: Allow at least EFS mounts to Fargate Containers #53

Open
archisgore opened this issue Dec 13, 2018 · 98 comments

Comments

@archisgore
Copy link

@archisgore archisgore commented Dec 13, 2018

Tell us about your request
Allow mounting of at least EFS volumes (if nothing more generic or extensible) onto Fargate tasks.

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We're in en-masse plans to migrate to Fargate. We're 100% in agreement with having stateless containers talking to stable external storage over the network (S3, DynamoDB). We'll do it sooner or later. You won't lose business without this.

This is an empathetic ask - if we could mount at LEAST EFS volumes to support those external workloads (stuff we don't build, but rather download), then it allows a large life-and-shift to Fargate, getting rid of Docker for AWS and ECS and gives us one consistent team-wide technology to consume, while we the factor out those dependencies cleanly.

Are you currently working around this issue?
We use Docker Swarm using the old DFA CloudFormation stack. Looked into ECS before volume plugins were supported and just the 2-3 level steps was awful (create volume, mount on host, remember where it is on host, launch task, mount directory to volume, mount volume to container.)

@archisgore archisgore added the Proposed label Dec 13, 2018
@FernandoMiguel

This comment has been minimized.

Copy link

@FernandoMiguel FernandoMiguel commented Dec 13, 2018

ECS now allows for volumes to be mounted at task level, not host only..
Check it out.

@casperc

This comment has been minimized.

Copy link

@casperc casperc commented Jan 8, 2019

@FernandoMiguel : Does that mean that EFS is now supported by Fargate?

@Akramio

This comment has been minimized.

Copy link

@Akramio Akramio commented Jan 8, 2019

Thanks everyone for this request. It would really be awesome if you could give us a little more detail about your need for this feature: For example, which workloads / applications that require EFS would you want to deploy on ECS? Would also love to hear about any potential use-cases or interests in using the newly released FSx file system.

@jonathonsim

This comment has been minimized.

Copy link

@jonathonsim jonathonsim commented Jan 9, 2019

We'd very much like to see support for EFS in fargate. I imagine there's a multitude of applications - the one we have is that we (Idealstack) are doing website hosting in ECS, and want to support common PHP-based web apps such as wordpress, drupal, peoples custom PHP code etc. These typically require shared storage if you want to cluster them and autoscale, and don't support S3 in general. So for instance the AWS reference architecture for wordpress, magento or drupal all use EFS

But I would imagine in any situation where users want persistant storage wth unix filesystem semantics EFS is going to be helpful, particularly where you are moving existing apps into fargate. There's a lot of frustration on the internet over the lack of EFS support in fargate dating to when it was first released. FSx filesystems aren't something we use, but would probably also have similar applications, as would EBS support. Something similar to the new volume driver support for (non-fargate) ECS would be great, even if only certain AWS-managed drivers that supported a few common targets such as EFS, EBS and/or FSx were supported

With EFS in fargate we could support more effective autoscaling of these kinds of apps compared to EC2-based architectures (since a container can boot in seconds versus minutes to create an instance and add a container instance in ECS). This would be a killer feature for our product. Lack of EFS support stops us from supporting Fargate at the moment though.

@lifeofguenter

This comment has been minimized.

Copy link

@lifeofguenter lifeofguenter commented Jan 9, 2019

adding to @jonathonsim -there are quite some open-source services that are not built cloud-native:

  • jenkins (the primary/master, not nodes)
  • grafana (or many other graphing tools)
  • logstash/beats

Many of those tools just need a persistent storage for minimal writes.

@archisgore

This comment has been minimized.

Copy link
Author

@archisgore archisgore commented Jan 9, 2019

As original author, I'll give you some my use-cases:

  1. Private Docker registries (unless ECR allows us to host public/private repos for external distribution) like Harbor or Nexus. They can store blobs to S3, but still need a filesystem for state/config.
  2. Other legacy software that reads/writes config/state to/from disk - Java Webservers, etc.
  3. SQL databases. I know, I know, Aurora and RDS, but just trust me on this. We are a cybersecurity company, so we need to occasionally host containerized databases to host wordpress against. EFS would allow the database to be persistent, but Fargate would allow us to test various SQL injection scenarios against it and mitigations.
  4. Wordpress plugins generate new PHP code on the fly. This can't go to S3. Can't go to DynamoDB. Needs to be persisted under a filesystem.

The counter-pressure on why NOT to use ECS: That escape-hatch becomes an opportunity for hard-work-creep. Opens up new AMIs, custom Linuxes, host drivers, firewalls, authentication, and more. It's too much opportunity opened up only to give someone the ability to mount stage storage so their wordpress plugin can generate code on the fly.

@ahammond

This comment has been minimized.

Copy link

@ahammond ahammond commented Jan 9, 2019

Our use case would be mounting a volume read-only which contains static data (in our case the reference genome) instead of having to put this data in the image or download it from S3 every time we start a container.

@peterfranzen

This comment has been minimized.

Copy link

@peterfranzen peterfranzen commented Jan 10, 2019

This would be extremely helpful. We have a task that we'd like to run in Fargate that currently involves pulling around 30GB of data in from S3 each time it runs; we can do this in EC2 or on ECS containers, but it would save us a ton of headaches if we could load it directly into a Fargate volume.

@abby-fuller abby-fuller added this to We're Working On It in containers-roadmap Jan 10, 2019
@abby-fuller abby-fuller added the ECS label Jan 10, 2019
@JoseRolles

This comment has been minimized.

Copy link

@JoseRolles JoseRolles commented Jan 15, 2019

Yes, this would be awesome! Even this AWS whitepaper on WordPress "best practices" in the "stateless web tier" references using EFS to store plugin files https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf

Exactly what we are trying to achieve with Fargate. This would allow [plugin/cms] updates to happen on the fly [by WP admins].

@soukicz

This comment has been minimized.

Copy link

@soukicz soukicz commented Jan 15, 2019

Exactly what we are trying to achieve with Fargate. This would allow updates to happen on the fly.

That is actually something that you don't want because than you don't hava atomic deployment.

But EFS in fargate does have use cases. For example shared file cache for compiled templates.

@JoseRolles

This comment has been minimized.

Copy link

@JoseRolles JoseRolles commented Jan 15, 2019

What we need is for WordPress admins to be able to add plugins on their own. Even the whitepaper suggests this in a "stateless web tier".

@AdrianAntunez

This comment has been minimized.

Copy link

@AdrianAntunez AdrianAntunez commented Jan 17, 2019

+1 here, we've been waiting for this feature since Fargate was launched. We have lots of applications with a high ratio of connections waiting to be migrated into Fargate to make use of autoscaling feature but we need to be able to mount the same EFS to these services in order to share some information between containers

@AdrianAntunez

This comment has been minimized.

Copy link

@AdrianAntunez AdrianAntunez commented Jan 17, 2019

ECS now allows for volumes to be mounted at task level, not host only..
Check it out.

That feature isn't supported currently by Fargate.

In addition @abby-fuller IMHO it should be labelled as Fargate.

@dambrogia

This comment has been minimized.

Copy link

@dambrogia dambrogia commented Jan 23, 2019

Having EFS + fargate would allow me to skim off 5-6 minutes of my deployments. It would also provide extra flexibility/agility for adjusting configs during triage time.

I could deploy to the EFS mount with the ability of instant upgrades/rollbacks through something like deployer rather than needing to build a new docker image + deploying that to an ASG/Cluster with hardcoded configs built into them and wait for the replace action to occur in cloud formation.

@cgswong

This comment has been minimized.

Copy link

@cgswong cgswong commented Jan 28, 2019

Our use cases involve running stateful workloads, such as cluster state (software requires file systems), custom databases, and CloudFoundry migration -- for workloads requiring file systems.

Would love this feature, though we are looking at a more mature EKS as well.

@skwokie

This comment has been minimized.

Copy link

@skwokie skwokie commented Feb 5, 2019

Hi, I'd like to ask if there is an ETA for this. Fargate is one of the platforms that we are looking into for migrating our service and NFS/EFS support is a very important feature that our service uses; and knowing the ETA will be very helpful for planning our schedule. Thanks.

@juultje123

This comment has been minimized.

Copy link

@juultje123 juultje123 commented Feb 8, 2019

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

@larryboymi

This comment has been minimized.

Copy link

@larryboymi larryboymi commented Feb 8, 2019

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

I'm doing something very similar with RStudio...

@juultje123

This comment has been minimized.

Copy link

@juultje123 juultje123 commented Feb 11, 2019

AWS says they are working on EFS support for ECS with Fargate, if I may believe this post: https://forums.aws.amazon.com/thread.jspa?messageID=816397&tstart=0

@mi-hol

This comment has been minimized.

Copy link

@mi-hol mi-hol commented Mar 9, 2019

this feature would allow for rehosting of existing applications with minimal effort. Desperately required!

@pauldraper

This comment has been minimized.

Copy link

@pauldraper pauldraper commented Apr 11, 2019

I would like to operate a FTPS/SFTP server with scalable/durable storage.

@AdrianAntunez

This comment has been minimized.

Copy link

@AdrianAntunez AdrianAntunez commented Apr 18, 2019

Do we have any ETA for that??

@teamfighter

This comment has been minimized.

Copy link

@teamfighter teamfighter commented Apr 18, 2019

We really need this feature. ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

@FernandoMiguel

This comment has been minimized.

Copy link

@FernandoMiguel FernandoMiguel commented Apr 19, 2019

ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

@teamfighter awful? was 4 lines of code for efs, and another 4 for the task definition

@teamfighter

This comment has been minimized.

Copy link

@teamfighter teamfighter commented Apr 19, 2019

@FernandoMiguel I didnt told that it's impossible. I told that it is uncomfortable solution.

@juultje123

This comment has been minimized.

Copy link

@juultje123 juultje123 commented Apr 19, 2019

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

@FernandoMiguel

This comment has been minimized.

Copy link

@FernandoMiguel FernandoMiguel commented Apr 19, 2019

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

pretty PLEASE don't use s3fs.... S3 is object storage... trying to treat it as persistent storage is a terrible idea @juultje123

@cmsops-jenkinsCI

This comment has been minimized.

Copy link

@cmsops-jenkinsCI cmsops-jenkinsCI commented Aug 28, 2019

I have liked the top of this post. I and multiple DevOps engineers at my company ($3.9billion educational technology firm) have been waiting for this since Fargate came out. Many use cases for persistent and/or shared volumes and S3 is just too slow to be a viable solution. We hope for a near-term solution to address need of shared volume for multiple applications currently managed in EC2/ECS with Lambdas for graceful auto-scaling needs. We have discussed the need with our TAM and will be escalating this request soon, as we really want to get 20+ applications already in production migrated to Fargate.

@guybumbling

This comment has been minimized.

Copy link

@guybumbling guybumbling commented Aug 28, 2019

Ditto all of the above....AWS Engineers should be able to see the plethora of use-cases for need of Shared Volumes in Fargate.

@willfarrell

This comment has been minimized.

Copy link

@willfarrell willfarrell commented Aug 28, 2019

You can see the current status here: https://github.com/aws/containers-roadmap/projects/1?card_filter_query=53
Currently: We're Working On It

@tsykora-verimatrix

This comment has been minimized.

Copy link

@tsykora-verimatrix tsykora-verimatrix commented Sep 9, 2019

Two of our services need to download files from s3, process them and then return them back to s3.
Those files are large, well beyond Docker / Fargate volume limits.
These are the two last services out of 30 that we couldn't Dockerize with the only reason being lack of persistent storage support in Fargate and are holding back release of 28 remaining services.
So this feature is essential for us.

@oliversalzburg

This comment has been minimized.

Copy link

@oliversalzburg oliversalzburg commented Sep 12, 2019

I was trying to get Nexcloud working on Fargate and wasn't aware of this limitation. So much wasted time 😢 I bet if I go another road now, the next day this feature will land.

@insanitybit

This comment has been minimized.

Copy link

@insanitybit insanitybit commented Sep 17, 2019

I'm trying to run an open source database on Fargate. The ability to have data stored locally is, obviously, critical. Fargate is dead in the water for me with its 10GB of storage and no upgradeable capacity.

@hjames9

This comment has been minimized.

Copy link

@hjames9 hjames9 commented Sep 17, 2019

@insanitybit

This comment has been minimized.

Copy link

@insanitybit insanitybit commented Sep 17, 2019

edit: Well, maybe so. It would solve my use case though. You're right that something like EBS would be better.

@dmeiser

This comment has been minimized.

Copy link

@dmeiser dmeiser commented Sep 17, 2019

Here is a use case where the persistent data is stored in S3: We use a commercial Git LFS application that is distributed as a container and using S3 as a storage backend. When doing a git lfs commit, the files must first be stored on the container filesystem before being moved to S3. We have single files larger than 10GB, which excludes us from using Fargate. This pattern is fine for a sufficiently large enough EBS volume on an ECS EC2 instance, but introduces maintenance and engineering overhead.

@aaronsturm

This comment has been minimized.

Copy link

@aaronsturm aaronsturm commented Sep 24, 2019

We run Atlassian's Jira self-hosted, so I'd prefer to use Fargate so that I don't need to manage the system. I'm currently using EC2 with EFS for that.

@jestrjk

This comment has been minimized.

Copy link

@jestrjk jestrjk commented Sep 24, 2019

I think the original vision for fargate was something a little more on the ephemeral side, but it also seems that the concept of managed host clusters for containers is just too much of a good thing to limit to ephemeral use cases. I'd host a lot more things on fargate if I could. Gitlab installations, Solr servers, you name it. If it has persistent storage needs, I host it somewhere I can do that, if it doesn't, it goes into fargate.

@mgangl

This comment has been minimized.

Copy link

@mgangl mgangl commented Oct 15, 2019

We use ECS and EC2 instances to process large data files (>10GB) as they come in. It would be nice to use Fargate and EBS or EFS volumes to do this without then having to worry about EC2 autoscaling for the times of downtime. Most, if not all, of our tasks are called from SWF and work nicely with the 'task' based processing.

Right now we can't use fargate (or lambda) for processing these large files and things sit idle when not being used.

@duartegarin

This comment has been minimized.

Copy link

@duartegarin duartegarin commented Oct 18, 2019

The lack of this feature means we will need to go with EC2 for ECS which is a shame as Fargate is exactly what we wanted. But the application is a Drupal CMS, and EFS is a must have.

@Non-PlayerCharacter

This comment has been minimized.

Copy link

@Non-PlayerCharacter Non-PlayerCharacter commented Oct 21, 2019

Does anybody know if this feature is like 2 months out, or 6 months out, etc?
Thank you.

@jestrjk

This comment has been minimized.

Copy link

@jestrjk jestrjk commented Oct 28, 2019

The lack of ECS/Fargate team feedback is a little disheartening. The roadmap board still just says "we're working on it" with no updates or information. The issue is fast approaching a year old, with very little activity or feedback provided from the AWS teams involved. Throw us some bones friends.

@nignatov

This comment has been minimized.

Copy link

@nignatov nignatov commented Oct 28, 2019

@jestrjk, totally agree with you.
On AWS Summit Warsaw (in May) I spoke about this feature with AWS Architects and I get the answer - "very soon".
This very soon is almost 5 months already so I would love to hear more because it is really important feature I believe (at least for me :D ).

@imthy

This comment has been minimized.

Copy link

@imthy imthy commented Oct 29, 2019

Please update ETA for this feature. It helps us to make some decisions.

@coultn

This comment has been minimized.

Copy link

@coultn coultn commented Oct 29, 2019

I can confirm that we are working on this feature. When it launches, ECS task definitions will include additional options for volumes. These will be available on both EC2 and Fargate launch types. The snippet below is not necessarily the final design but it is representative of the current thinking - please let us know if you have questions or comments! When we get closer to launch, we will move the github issue to 'coming soon'.

(Note: the transitEncryption and readOnly fields are optional).

{
    "family": "my-task-with-efs",
    "volumes": [
        {
          "name": "myEfsVolume",
          "EFSVolumeConfiguration": {
                "filesystem": "fs-1234",
                "rootDirectory": "/path/to/my/data",
                "transitEncryption": "tls"
            }
        }
    ],
    "containerDefinitions": [
        {
           "name": "container-using-efs",
           "mountPoints": [
               {
                   "sourceVolume": "myEfsVolume",
                   "containerPath": "/mount/efs",
                   "readOnly": true
               }
        ]
    }
   ]
}
@david-sitsky

This comment has been minimized.

Copy link

@david-sitsky david-sitsky commented Oct 29, 2019

That is good news. I assume EBS support will be a part of this work?

@coultn

This comment has been minimized.

Copy link

@coultn coultn commented Oct 29, 2019

That is good news. I assume EBS support will be a part of this work?

EBS support is not included in this feature - this is EFS only.

There is another issue for EBS support: #64

If you are interested in EBS support, please comment on that issue, thanks!

@borgstrom

This comment has been minimized.

Copy link

@borgstrom borgstrom commented Oct 29, 2019

I’m so happy to see this is planned for both EC2 and Fargate tasks!

I assume the rootDirectory is a path inside the EFS volume and not where to mount the volume on the host (since you wouldn’t know where to mount things on Fargate). If so, out of curiosity, where does the EFS volume get mounted on EC2 hosts?

@petderek

This comment has been minimized.

Copy link

@petderek petderek commented Oct 30, 2019

@borgstrom Correct, similar like this:

mount -t efs fs-abcde:${rootDirectory} /path/to/local/mount

If so, out of curiosity, where does the EFS volume get mounted on EC2 hosts?

We're still working out the details, but the pattern should feel familiar. We'll create an nfs/efs mount on the host that can then be mounted into the container(s) that use it.

@hjames9

This comment has been minimized.

Copy link

@hjames9 hjames9 commented Oct 30, 2019

Will EBS/local persistent volumes be included with this EFS change?

EBS is something different than EFS. EBS will be never supported by fargate.
EFS is network persistent volume

@bordeux Looks like you're wrong about this.
#64

@melaraj2

This comment has been minimized.

Copy link

@melaraj2 melaraj2 commented Nov 1, 2019

EFS is so slow and limited and expensive. We really need support fora selection of volume plugins such as netshare for SMB shares, EBS, SSHFS.

@tianmarin

This comment has been minimized.

Copy link

@tianmarin tianmarin commented Nov 12, 2019

EFS is so slow and limited and expensive. We really need support fora selection of volume plugins such as netshare for SMB shares, EBS, SSHFS.

I thought @melaraj2 was shitposting, but we have to be honest; EFS performance is not ideal to many projects:

EFS is up to three orders of magnitude slower than the EBS counterpart from which you want to migrate [Lawrence McDaniel]

Anyway, allowing Fargate containers to mount EFS filesystems is a HUGE progress, even just to execute tasks within the EFS filesystem.

Imagine you want to backup the EFS filesystem syncing it to a S3 Bucket (or execute any periodic tasks inside the EFS content). Instead of, having to imagine how to execute this task on one of the hosts that might (or might not) mount the EFS, or fire up a EC2 Scaling Group to mount the FS and execute the tasks and terminate;

Execute this kind of tasks on containers, feels more natural.

@melaraj2

This comment has been minimized.

Copy link

@melaraj2 melaraj2 commented Nov 12, 2019

Anyway, allowing Fargate containers to mount EFS filesystems is a HUGE progress, even just to execute tasks within the EFS filesystem.

Imagine you want to backup the EFS filesystem syncing it to a S3 Bucket (or execute any periodic tasks inside the EFS content). Instead of, having to imagine how to execute this task on one of the hosts that might (or might not) mount the EFS, or fire up a EC2 Scaling Group to mount the FS and execute the tasks and terminate;

Execute this kind of tasks on containers, feels more natural.

We actually tried to use EFS. We have hundreds of thousands of small files 1 to 5KB, and the backup process was taking longer than a day. It was impossible to use, and even if you increase the throughput, it made very little difference, and because the pricing is set by throughput consumption, it was crazy expensive.

For Fargate to be complete, it needs the ability to do storage plugins, most importantly EBS, and since I am venting, why is EBS still only in one AZ, it's over 10 years old, and AWS has storage technology in Aurora that spans zones, when is EBS going to cross zones. It would really help deployment of highly available systems like MongoDB.

@hudsondba

This comment has been minimized.

Copy link

@hudsondba hudsondba commented Nov 13, 2019

Allowing Fargate containers to mount EFS filesystems it's quite nice for who are deploying Apache Airflow into Fargate. You can easily share your dags into containers using the same filesystem. Without EFS and using Fargate it requires an extra container with Mount Points to share the files.

@ericandrewmeadows

This comment has been minimized.

Copy link

@ericandrewmeadows ericandrewmeadows commented Nov 14, 2019

@hudsondba - The other option is to use cron to sync the DAGs from S3.

@hudsondba

This comment has been minimized.

Copy link

@hudsondba hudsondba commented Nov 14, 2019

@ericandrewmeadows It's true but it's not a good practice to run cron jobs inside docker containers. Today I'm solving this problem with another container inside the same task definition sharing the same volume and syncing with GitHub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
containers-roadmap
We're Working On It
You can’t perform that action at this time.