efs helper fails with "no such device" on Amazon Linux 2 docker volume, yet nfs works

[diablodale]

Hi. I've isolated a scenario where the EFS mount type helper fails to create a docker volume, yet using the NFS mount type directly is successful. 🤔 The error is no such device. I suspect this is related to the addr= NFS mount option.

Setup

• Amazon Linux 2 currently with kernel 4.19.75-28.73.amzn2.x86_64
• ECS optimized AMI having docker 18.09.9-ce
• ECS agent/init 1.36.1
• amazon-efs-utils 1.18

---

Repro of Failure

1. Create your VPC, security groups, NACLs, etc.
2. Create an EFS filesystem, encrypted with default key, general purpose, bursting.
3. Note the EFS fs id. For writing this repo, I will use fs-12345678
4. Create EC2 instance from an Amazon ECS-optimized AMI
5. Create your ECS cluster with that instance. (this step is probably not needed)
6. SSH into the EC2 instance
7. Create a docker volume

   docker volume create \
       -d local \
       -o "type=efs" \
       -o "device=fs-12345678:/" \
       -o "o=tls" \
       myefs

8. Note there is no error.
9. Run a docker container that mounts this volume into the container's filesystem

   docker run -ti -v myefs:/mnt/one alpine sh

Result

The container does not run ☹ and the below no such device error is given:

docker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/myefs/_data': failed to mount local volume: mount fs-12345678:/:/var/lib/docker/volumes/myefs/_data, data: tls: no such device.

Expected

No error. Running container. And the EFS filesystem is mounted into the container's /mnt/one.

---

Repro of Success with NFS

1. docker container prune and yes
2. docker volume rm myefs
3. Run

   docker volume create \
       -d local \
       -o "type=nfs" \
       -o "device=:/" \
       -o "o=addr=fs-12345678.efs.us-east-1.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport" \
       myefs

4. Note no error.
5. Run a docker container that mounts this volume into the container's filesystem

   docker run -ti -v myefs:/mnt/one alpine sh

Successful Result with NFS

Yay. 👍 You are now in the running Alpine container and the EFS filesystem was successfully mounted to the container's /mnt/one

Please note the params on the volume creation. I used the addr= param instead of putting the DNS on the device.

---

Repro of FAILURE with NFS

Below is a failure scenario using the typical NFS parameters, where the filesystem DNS is on the device and does not use the addr= parameter. This approach is what appears in the EFS console itself.

1. docker container prune and yes
2. docker volume rm myefs
3. Run

   docker volume create \
       -d local \
       -o "type=nfs" \
       -o "device=fs-12345678.efs.us-east-1.amazonaws.com:/" \
       -o "o=nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport" \
       myefs

4. Note no error.
5. Run a docker container that mounts this volume into the container's filesystem

   docker run -ti -v myefs:/mnt/one alpine sh

Failure Result with NFS

The container does not run ☹ and the below invalid argument error is given:

docker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/myefs/_data': failed to mount local volume: mount fs-12345678.efs.us-east-1.amazonaws.com:/:/var/lib/docker/volumes/myefs/_data, data: nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport: invalid argument.

Expected Result with NFS

No error. Running container. And the EFS filesystem is mounted into the container's /mnt/one.

[diablodale]

Related info...
AWS support article https://aws.amazon.com/premiumsupport/knowledge-center/ecs-create-docker-volume-efs/
PR merged into Docker moby related to type=nfs and that addr param https://github.com/moby/moby/pull/27329/files

[lilferna]

Thanks for the feedback @diablodale. We’ll have someone take a look and post a more detailed response once we better understand the issue.

[lilferna]

The no such device error you are seeing is happening because the EFS mount helper is not compatible with the docker local volume plugin. Currently the only way to reference an EFS file system in your task definition is to mount using NFS. This can be done using the method you specified as working, or using the preview integration between ECS and EFS (link). When the ECS/EFS integration becomes generally available, it will leverage the EFS mount helper, thus supporting TLS encryption, IAM authorization, and EFS access points. You can track status of that roadmap item from the container roadmap (link). For now, if you require TLS, IAM, or Access Points, a workaround would be to pre-mount the file system using the EFS mount helper on the ECS hosts, and then use directory bind mounts to pass the paths through to the container.

[diablodale]

I agree there is errant behavior. However, you have collapsed EFS and ECS together. Instead, the EFS incompatibility is independent of ECS -- is it distinct.

EFS can not be securely used (data encrypted in transit) on EC2 instances that use docker containers and industry standard deployment tech like docker-compose that manage volumes as abstracted stores. Pre-mounting hacks are naturally unsustainable as it is the role of deployment tech to do such mounting.

Does your team know specifically the bug (limitation, etc.) with the EFS helper that causes the failure? What diagnostics or design flaws can you share? If nothing is known, I request your EFS team investigate this errant behavior. Fix this and you please both your EC2 and your ECS customers because with a fix, both of them can use it. (The narrow ECS work being done suggests you know customers want this, but making that ECS only is...limiting)

What are the possibilities here?

[lilferna]

Thanks for your feedback. The root cause for the current behavior is known, and we are tracking the ability for ECS containers to mount EFS using TLS encryption with this roadmap item: aws/containers-roadmap#53.

[diablodale]

Gosh, you wrote the same thing w/ same link you already wrote. Confused. 🤔

Or perhaps are you providing a non-answer? Is your answer actually "We know the EFS issue, and we won't fix it except for ECS customers"?
If that's your answer, please just state it and lets move on.
But if that's not your answer, please consider my inquiry about EC2 customers (not ECS).

[wochanda]

Got it, we thought you were using ECS because in your initial post you provided your ECS agent version. The only supported way to mount EFS from bare Docker is via NFS using the local driver, since this driver makes mount syscalls rather than calling the 'mount' command directly, and the EFS mount helper only works when called in the latter way. We don't have anything to share at this time on releasing a Docker volume plugin for EFS.

[diablodale]

Thanks for walking me though that. I get both the reason for the bare Docker fail, and AWS's current stance.
Please keep in futuremind the request for a richer (e.g. mount syscall) EFS mounting solution.
I have no other questions at this time.

[gitmlynch]

EFS can not be securely used (data encrypted in transit) on EC2 instances that use docker containers and industry standard deployment tech like docker-compose that manage volumes as abstracted stores. Pre-mounting hacks are naturally unsustainable as it is the role of deployment tech to do such mounting.

@diablodale, do you feel that there the pre-mounting hack described is insecure? We've run into this same problem and are considering using pre-mounting as a workaround. We don't use docker-compose. Instead we use a collection of python scripts in an autopilot pattern for the instances to configure themselves. We can edit those scripts to do the host pre-mount and adjust the docker bind mounts as needed.

[diablodale]

hi. Back in 2020, it was possible to manually mount EFS with the custom tools AWS provided -- those custom tools supported encryption in transit. In contrast, if standard tools (e.g. standard mount helpers/api) were used, it was not possible to encrypt in transit. The AWS team post here spoke to that #39 (comment)

Its been more than a year and likely much has changed. I think I heard (unconfirmed) that AWS now supported standard tools/apis and docker added yet better support for AWS and its volumes/filesystems. I suspect a better story if I were to revisit this with my solutions today.

Good luck 👍