Volume driver support

[robhaswell]

The VolumeDriver parameter is now part of the Docker Remote API 1.21. It would be great to get this included in the ECS task definition. This would allow me to access more storage options for my Docker containers, for example, accessing EBS volumes through Flocker or using the Ceph RDB driver (http://www.sebastien-han.fr/blog/2015/08/17/getting-started-with-the-docker-rbd-volume-plugin/).

[wallnerryan]

@euank what is the delta to minimally support these changes? Would assume that the ecs-agent would recognize the the host volume doesnt need a sourcePath, but rather could be just a name. Also the container definition or task definition could take a new volume driver attribute. This would match parity with docker volume create --volume-driver=driver --name=vol_name for 1.9 and docker run --volume-driver=driver <name>:/path/

Looking into making some changes

[robhaswell]

Docker also has additional error conditions associated with volume drivers. Search your souls for if this is MVP.

[kapilt]

We're also very interested in this support

[ColinHebert]

Any news regarding the support of volume drivers?

[ferrantim]

+1 would love to see this. Getting more and more requests.

[brianantonelli]

+1 this would be great

[analogrithems]

+1 this is holding up a lot of use cases and google compute already makes this available

[ferrantim]

@cbbarclay any update on when ECS will support Docker volume drivers? Thanks!

[robhaswell]

I recently evaluated deploying some new infrastructure on ECS but this prevented me from doing so.

[p7k]

+100500

[ryansch]

It's a new record! 😉

[sebalas]

👍

[gservat]

+1 !!

[siimleetberg]

+1

[ebuildy]

Yeah! will be useful to make possible to use EFS shares via the plugin http://netshare.containx.io/

[ervinb]

+1

[joshhiggins22]

+1

[ghost]

+1. An EFS/NFS volume driver is a critical requirement for many use cases. Could be http://netshare.containx.io/ (which is not yet production ready but quite nice) or a new one designed with EFS in mind

[egafni]

+1 EFS/NFS

[ainagy]

+1

[RJSzynal]

+1. I want to use netshare.containx.io too

[ebirukov]

+1

[balopat]

+1

[JaimeVL]

+1. This will enable ECS to seamlessly support stateful applications, which is a big gap in my opinion.

[rafagnin]

+1

[kislyuk]

+1

[mounemoi]

+1

[marcovzla]

+1

[myedibleenso]

+1

[kchilder]

At least wrap the nfs volume driver so we can use it with EFS!

[galindro]

@euank @samuelkarp @aaithal @richardpen

This feature request was opened at Oct 26 2015. Almost 2 years have passed and nothing about it was discussed. IMHO, this is a little feature that could be easily be developed by AWS team.

Please, someone from AWS could give us an ETA or just something about if this feature is in the roadmap for this month, or this year?

[galindro]

Guys, I've a workaround for this to work.

I'm using Docker Cloudstor as a plugin for persistent storage. To make it work in Amazon ECS, I've did the following steps:

1. Install it under the container instance. This instalation doesn't supports EFS. If you want EFS support, you need to change the plugin installation command line. Check the Docker for AWS official template to get the command line.

docker plugin install \
  --alias cloudstor:aws \
  --grant-all-permissions \
  docker4x/cloudstor:17.06.0-ce-aws2 \
        CLOUD_PLATFORM=AWS \
        AWS_REGION=$INSTANCE_REGION \
        EFS_SUPPORTED=0 \
        DEBUG=1

2. If the container instance is an Amazon ECS Optimized AMI, you must create /dev/mqueue directory, as described here.

mkdir /dev/mqueue

3. Create a volume. ATTENTION: this is a relocatable volume. If you must share it with more than one task, please use shared volume instead, but you should enable EFS first by passing the correct parameters to the plugin install command line.

docker volume create -d "cloudstor:aws" --opt ebstype=gp2 --opt size=1 --opt backing=relocatable myvol

4. Add the volume to your task definition:

5. Add the mountpoint to the container:

6. Update the service and inspect the container.

7. Go drink a good beer, preferably an Indian Pale Ale.

[samuelkarp]

@galindro While we don't generally comment on our future roadmap (including ETA), I can shed a bit more light onto the concerns here.

We've generally been thinking about volume drivers in terms of application persistent storage. There are a few different models where persistent storage can be used for applications:

• Shared storage, where each copy of your application sees the same files. One of the more common technologies for this is NFS, and the Amazon EFS service provides a hosted version of NFS. In terms of scheduling, it doesn't matter how many copies you would have as long as they could all see the same data.
• Pre-populated, independent storage. This would encompass a situation where each copy of your application wants to see the same data-set at start, but does not need to share mutations. Common use-cases for this involve things like read-only views or copy-on-write; Amazon EBS provides a hosted block storage service with volumes created from snapshots. In terms of scheduling, it doesn't matter how many copies you have as long as they could all start from the same data.
• Persistent, restartable storage. This would be a use-case akin to upgrading a traditional stateful application in-place, but with a container wrinkle added. Amazon EBS provides a hosted block storage service where volumes can be attached and detached from instances. In terms of scheduling, you'd only have one copy here with its own storage. If you were to have multiple copies of the application, each would see separate data.
• Persistent, sharded storage. This would be a use-case akin to the previous one, except that multiple replicas might each see different shards. In terms of scheduling, we'd have to maintain a mapping of shard-to-application-copy to ensure that storage doesn't get lost or unused; complications would occur during deployments or during scale-out/scale-in.

Volume drivers could be paths to enabling all of these use-cases. However, there are some challenges with volume drivers specifically:

• Many of the clustered volume drivers require all of your instances to be able to talk to each other and be configured together. We'd need to figure out how to handle error conditions where this isn't the case, so that you have enough information in order to debug it.
• The volume drivers don't indicate to us what their scheduling requirements are: are we free to start as many copies as we want (desired count in a service), are we restricted to only running one, do we need to handle a persistent identity that gets passed in to a given container configuration, etc.

[frimik]

I just need "blocker" type of storage in the first place... With a few extra options maybe. I can get around all of it with "magic" hints in the definitions / the ECS ui. But for how long do we have to keep doing it that way... :).

…

On tor 21 sep. 2017 01:35 Samuel Karp ***@***.***> wrote: @galindro <https://github.com/galindro> While we don't generally comment on our future roadmap (including ETA), I can shed a bit more light onto the concerns here. We've generally been thinking about volume drivers in terms of application persistent storage. There are a few different models where persistent storage can be used for applications: - Shared storage, where each copy of your application sees the same files. One of the more common technologies for this is NFS, and the Amazon EFS service provides a hosted version of NFS. In terms of scheduling, it doesn't matter how many copies you would have as long as they could all see the same data. - Pre-populated, independent storage. This would encompass a situation where each copy of your application wants to see the same data-set at start, but does not need to share mutations. Common use-cases for this involve things like read-only views or copy-on-write; Amazon EBS provides a hosted block storage service with volumes created from snapshots. In terms of scheduling, it doesn't matter how many copies you have as long as they could all start from the same data. - Persistent, restartable storage. This would be a use-case akin to upgrading a traditional stateful application in-place, but with a container wrinkle added. Amazon EBS provides a hosted block storage service where volumes can be attached and detached from instances. In terms of scheduling, you'd only have one copy here with its own storage. If you were to have multiple copies of the application, each would see separate data. - Persistent, sharded storage. This would be a use-case akin to the previous one, except that multiple replicas might each see different shards. In terms of scheduling, we'd have to maintain a mapping of shard-to-application-copy to ensure that storage doesn't get lost or unused; complications would occur during deployments or during scale-out/scale-in. Volume drivers could be paths to enabling all of these use-cases. However, there are some challenges with volume drivers specifically: - Many of the clustered volume drivers require all of your instances to be able to talk to each other and be configured together. We'd need to figure out how to handle error conditions where this isn't the case, so that you have enough information in order to debug it. - The volume drivers don't indicate to us what their scheduling requirements are: are we free to start as many copies as we want (desired count in a service), are we restricted to only running one, do we need to handle a persistent identity that gets passed in to a given container configuration, etc. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#236 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXZzaa-S9eY8cpSNv8zexvvFZjMMMEtks5skaFQgaJpZM4GVrUW> .

[robhaswell]

@samuelkarp I understand your concerns but I don't think they are great enough to prevent this feature from being considered for work.

• Many of the clustered volume drivers require all of your instances to be able to talk to each other and be configured together.

This is true, however for the simplest (and most obvious?) use-case of using the EBS volume driver, this is not a concern.

• The volume drivers don't indicate to us what their scheduling requirements are

They do, they pass back an error (I think, it's been two years since I looked at this) which then indicates to ECS that the container is unschedulable and requires intervention by an administrator. This is how Kubernetes handles the problem. The solution is probably for the user to not use a storage provider with inflexible scheduling requirements - again, EBS does not have this problem (probably... is the 26 volume-per-instance limit still in place?)

I understand the desire to not expose users to features that might break as a general concept. Mobile phones, cable boxes, Echo-devices... these are all things which should not come with sharp edges. AWS, however, is a smart platform for smart people. Infrastructure is hard. I think AWS should give its users the benefit of the doubt and allow them enough rope to hang themselves. To that I mean, this feature has caveats for sure but the benefits outweigh the negatives, because right now the alternative is to move your workload to a scheduler which does support storage services... like.. GKE :(

[samuelkarp]

I understand your concerns but I don't think they are great enough to prevent this feature from being considered for work.

My apologies, I did not intend to imply that these concerns are preventing this feature from being considered. I just wanted to give a little more information as to the type of things we're thinking about.

They do, they pass back an error (I think, it's been two years since I looked at this) which then indicates to ECS that the container is unschedulable and requires intervention by an administrator

An error like that is a very late-binding failure, occurring after a placement decision has taken place. Failures are definitely a signal that can be taken into account by a scheduler, but I was talking about information that can be exposed prior to a placement decision being made.

[galindro]

In the meanwhile, I think that IT admins/Devops could use my workarround considering that almost all storage driver for Docker exposes the volumes in a way that they could be used by docker run just referring them by name, like this:

docker run -v myvolume:/foo -w /foo -i -t ubuntu bash

The only problem that this approach have, IMHO, is that you need to create a volume first with the docker volume create command.

I've tested with Docker Couldstor, but I think that it would work with RexRay.

@frimik I think that blocker is a little old solution. Try Docker Couldstor. It is a better solution.

[rclark]

@samuelkarp I want to point out another use-case that I'm not sure is covered by your bullets above. In our applications, data is persisted elsewhere (S3) and updated regularly by separate systems. Our ECS containers need to start from the current state of that data, perform their processing, and write the output elsewhere.

• the data's starting state varies from container to container
• the data on disk that the container generates is ephemeral

In this case the most important missing features are accounting for disk space during scheduling, and limiting disk usage to some reserved amount. Limiting can be accomplished via volume driver options.

Aside from the accounting issues, all we'd need for limiting would be the ability to pass arbitrary --volume-opts through the TaskDefinition.

[Stavanger75]

+1

[dsnopek]

Volume drivers are a pretty critical feature of Docker - it seems weird not to allow us to use them?

[CameronGo]

@samuelkarp while I understand there is some appetite for broader and more complex volume driver support, does it seem likely that ECS would allow the mounting of an EBS volume and/or an EFS data store as part of the Task Definition at some point in the near future? My guess is that addresses the vast majority of the use cases right?

[gitrc]

@samuelkarp I know Amazon position is to not disclose the roadmap and whatnot but can you please give us an update on this much needed functionality? I saw the announcement for ENI to container (awsvpc) which solves the networking equivalent of the volume driver ask in this issue. The ability to associate an EBS volume would go a long way. Glad to see ECS feature development is active.

[csumpter]

+1 I poorly assumed this feature would have been present in the task definition features - and was banking on being able to use it. It would be great addition!

[kislyuk]

👍 this really puts a damper on our use of AWS Batch (which uses ECS).

[wdalmut]

👍

[DaveWK]

This really makes working with efs and ecs difficult. The fact that overlay2 is not supported with nfs4 (due to d_type) in docker means you can't simply change your docker graph location for docker daemon. I had hoped the alternative would be using a log driver, but this is a mess if I can't make unique volumes per task.

Alternatively if I could even specify a dynamic value in my volume bind (so each task had it's own "writer" volume) it would make efs and ecs much more usable.

[yhlee-aws]

We have launched support for Docker volume driver: https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-ecs-now-supports-docker-volume-and-volume-plugins/

This feature is available through amzn-ami-2018.03.d-amazon-ecs-optimized AMI, or from agent v1.20.1

You can find the developer guide here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

Please let us know if you have any questions.

[wmitsuda]

Does this not work with Cloudformation yet?

[yhlee-aws]

CFN support for this feature isn't available yet, but we are working with CFN team to enable it.

[analogrithems]

@yunhee-l any chance you could provide a task def example or tutorial for using docker volume driver for efs or ebs? The links you provided are just marketing links and the developer guide just states it's available but no details on how.

[CameronGo]

@analogrithems - i spent some time working with it this evening and was able to get a persistent EBS volume attached to a task launched from ECS. I'm no expert at this point, but here are the high points:

1. Update your ecs agent to the latest version (ours runs on Ubuntu in a container so just required a docker pull and then stop/remove/start the agent again
2. Install a volume plugin driver. There maybe others or better out there, but I did this by running

docker plugin install rexray/ebs

3. Make sure the EC2 instance's role has sufficient access to create/list/update EBS volumes. (In my test env I gave it temp full EC2 access)
4. At this point, manually running a container from docker CLI should let you run and connect to a newly provisioned EBS vol:

docker run -ti --volume-driver=rexray/ebs -v test:/test busybox

5. At this point a Task Definition with a volume configured should work. An excerpt from my Task Def volumes section looks like this:

  "volumes": [
    {
      "name": "test",
      "host": null,
      "dockerVolumeConfiguration": {
        "autoprovision": false,
        "labels": null,
        "scope": "shared",
        "driver": "rexray/ebs",
        "driverOpts": null
      }
    }
  ]