Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fargate] [request]: Allow to increase container disk space #384

Closed
Xantrul opened this issue Jun 21, 2019 · 28 comments
Closed

[Fargate] [request]: Allow to increase container disk space #384

Xantrul opened this issue Jun 21, 2019 · 28 comments
Assignees
Labels
Fargate AWS Fargate Proposed Community submitted issue Work in Progress

Comments

@Xantrul
Copy link

Xantrul commented Jun 21, 2019

Tell us about your request
It looks like disk volume one gets in Fargate is ~10 GB. It would be helpful to be able to configure the size it in Task/Container configuration. In our case, we would like to go as high as 150 - 200 GB.
(link to doc confirming this limit: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_limits.html)

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We have a service with a fairly simple flow: It downloads a zip file, extracts useful information from the file and publishes it into external database. The problem is - those zip files can be up to 100 GB in size.

The full flow we tested is: messages come into SQS queue -> Lambda reads from SQS and invokes Fargate task -> Fargate does the job

The workload is very spiky - sometimes we have dozens of files coming in at the same time and sometimes we have nothing for a day or two. So Lambda+Fargate combination works perfectly for us because we process jobs as soon as we get them without any wait, but at the same time don't pay for any resources when we have nothing to process. And nothing to manage too!

Are you currently working around this issue?
Old version of our service runs on multiple EC2 instances. For the new version we're currently evaluating options and since Fargate cannot be used looking into EKS.

Additional context
From other requests in this area it looks like you're already working on the option to attach EBS/EFS volumes, but in our case we don't need persistent storage - we just need more space which is created and destroyed with the container.

@Xantrul Xantrul added the Proposed Community submitted issue label Jun 21, 2019
@coultn coultn added the Fargate AWS Fargate label Jun 21, 2019
@timobehrens
Copy link

timobehrens commented Aug 6, 2019

I second that! I really enjoy using ECS with Fargate. Not having to manage the underlying infrastructure is really fantastic.

Tell us about your request
We have worker images, that have big machine learning models in them. We want to scale up and down based on demand and it would make our lifes so much easier if we can use Fargate for that.
The docker image size is currently over 15GB, running into the container limits.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We want to easily scale up and down based on demand. Currently we prefer to use managed services as much as possible, to minimize the management overhead. While we could use ECS with EC2, Fargate is the preferred option at this point.

Are you currently working around this issue?
Currently evaluating the options - maybe going for ECS with EC2.

@whereisaaron
Copy link

Other requested ways to increase Fargate storage include mounting EFS or mounting S3 storage
Ref #53 #412

@Xantrul
Copy link
Author

Xantrul commented Aug 6, 2019

Other requested ways to increase Fargate storage include mounting EFS or mounting S3 storage
Ref #53 #412

Yes, but for our use case ability to increase disk space directly would be preferred because (a) it is less components to provision/manage/delete and (b) it is unclear what would read/write speed of the attached EFS/S3 volume look like.

@haslam22
Copy link

We would also be interested in this. Our use case would be deploying and running a large bundled application (15GB+) via our Jenkins CI on ECS/Fargate. Currently we are too limited on space to achieve this.

@dancallan
Copy link

Yes, we too have a use case for this where we are currently performing some ingest tasks in fargate on new media files as they are uploaded to an s3 bucket and the existing (small) storage limit is restricting the sizes of files we can process.

Our image file is quite small. We basically just need slightly larger (ephemeral) storage to download the file to and output derivative thumbnails to before saving them to s3.

Pretty much exactly the same scenario as the OP (Xantrul).

The ability to choose from predefined storage options (just like cpu and memory is for fargate) would be a perfect implementation for this. We already choose task memory allocation based on the input file size and it would be trivial to also spec the storage allocation in the same logic.

@jca-agnitio
Copy link

Thumbs up for this request. We are using Fargate and love it.

We would like to be able to increase storage over 10Gb or even map task storage to S3.

@soapergem
Copy link

soapergem commented Jan 2, 2020

I was linked to this issue by AWS' support staff when I asked them how to increase the volume size for a Fargate container. Needless to say, I was equally disappointed that this isn't already an option. In my case I was writing a Step Functions state machine that transforms data through multiple phases (it launches Glue Scripts, Lambda functions, and yes - Fargate containers). I ran this pipeline today and quickly bumped into the disk space limit for one of the Fargate containers.

We have a running EKS cluster for some related work as well, so I could write it as a Kubernetes job and then use a Lambda to launch it, and make use of the Step Functions SDK to monitor progress. Or, I could write a bunch of Terraform to convert my ECS cluster into a managed one instead of relying on the defaults and ease of use that Fargate provides. There are probably other ways to go about it, too.

But regardless of which alternative path I choose, it's adding several days to my development cycle, when I was expecting to just be able to set a property in my task definition and be done with it. I am very surprised this isn't supported.

@Jamalarm
Copy link

We are also running into this issue. Very surprised and annoyed that there is a hard limit here.

@JoelAnalyst
Copy link

We have this problem and it is quite blocking. Could we know if you're working on it please ?

@zmarouf
Copy link

zmarouf commented Jan 23, 2020

Same thing here. Fat container image on EKS. Fargate node can't pull.

@shgolla
Copy link

shgolla commented Mar 9, 2020

Same with us, we have containers that needs to run machine learning models and would need storage more than 15GB. Currently tasks are failing with error
"CannotPullContainerError: failed to register layer: Error processing tar file(exit status 1): write /app/data/model_title_dmm_concat_1.trainables.syn1neg.npy: no space left on device"
Increasing this limit will be great. Any idea when is this planned to be addressed?

@rpgreen
Copy link

rpgreen commented Apr 13, 2020

+1

@michal-lyson
Copy link

michal-lyson commented Apr 13, 2020

Probably still not enough for most of use cases but fargate platform version 1.4.0 offers now 20GB storage space: https://aws.amazon.com/blogs/containers/aws-fargate-launches-platform-version-1-4/

@restfulhead
Copy link

And it's now possible to mount EFS volumes for additional storage: https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-ecs-aws-fargate-support-amazon-efs-filesystems-generally-available/

@zivyatziv
Copy link

Can we extend this to EC2 type also?
It can be generally applied using --storage-opt size=<size> while performing the docker run command, as stated here

@cweidinger
Copy link

We'd like a configurable amount of disk space above 20gb on Fargate on EKS.

@mwarkentin
Copy link

Looks like this is launching soon! https://github.com/aws/copilot-cli/releases/tag/v1.6.0

@vibhav-ag vibhav-ag self-assigned this May 2, 2021
@vibhav-ag
Copy link

https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ecs-aws-fargate-configure-size-ephemeral-storage-tasks/ This feature is available on ECS now.

@alanprot
Copy link

Is there any expected date to add support to this on CDK?
https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs.TaskDefinition.html

@markusa380
Copy link

@alanprot For now, you can do this:

  ephemeralStorageHack(taskDef: ecs.FargateTaskDefinition): void {
    const props = taskDef.node.defaultChild as ecs.CfnTaskDefinition;
    props.addPropertyOverride("EphemeralStorage", {
      SizeInGiB: 100,
    });
  }

@iwarshak
Copy link

@markusa380 your solution works, but after a deployment with CodePipeline/ECSDeployAction the EphemeralStorage override is lost. Have you experienced this?

@markusa380
Copy link

@iwarshak No sorry, I didn't use CodePipeline however.

@virtualbeck
Copy link

After some time trying to get this into my task definition json, I found that terraform supports it. Here's the correlating portion of the task definition for 200GiB ephemeral storage (FARGATE):

{
    "targetId": null,
    "targetType": null,
    "value": null,
    "name": "ecs.capability.extensible-ephemeral-storage"
}

@jonathanmarsaud
Copy link

jonathanmarsaud commented Apr 12, 2022

Hi,

Your link is only valid for ECS on Fargate, not EKS on Fargate.
Please reopen this issue, as we're also concerned and affected by this issue (it mades EKS on Fargate useless for our workflow due to this only issue).

Thanks

@JasonRD
Copy link

JasonRD commented Oct 10, 2022

emm.. Didn't it any progress in THREE years on EKS Fargate? @tabern

@gj
Copy link

gj commented Feb 21, 2023

@vibhav-ag if I would like to request an increase in Fargate ECS task size beyond the new 200 GiB limit, is this a good ticket to reopen or shall I create a new one?

@vibhav-ag
Copy link

@gj I would recommend opening another one with some detail on what is the driver for the ask. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Fargate AWS Fargate Proposed Community submitted issue Work in Progress
Projects
None yet
Development

No branches or pull requests