[ECS] [Fargate Task Storage]: Allow permission configuration of Fargate bind mounts

[Alex-Richman]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem:
Ephemeral storage for Fargate tasks with readonlyRootFilesystem and a non-root user.

On Fargate platform 1.3.0 this was achievable (in an undocumented/unintentional manner) by configuring a docker local volume at the task level and mounting it to /tmp in each service:

      volumes:
        - name: "tmpfs"
          dockerVolumeConfiguration:
            scope: "task"
            driver: "local"

...

          mountPoints:
            - sourceVolume: "tmpfs"
              containerPath: "/tmp"

(resulting in a world-writable tmp directory mounted to /tmp/ within the container)

On Fargate platform 1.4.0 docker local volumes are completely unavailable, and the new (officially recommended) way of implementing ephemeral storage for Fargate tasks is using a bind mount [1].

The problem with using a bind mount is that ECS mounts it as writable only by root, so a container running as a non-root user is unable to write any temporary files. Having the container run as root is generally undesirable for security reasons, though practically I expect the impact on ECS is limited since a root-based container escape would just dump an attacker into the ECS host which is presumably heavily sandboxed.

The ideal solution would be for ECS to support configuring permissions on bind mounts, or better still support tmpfs on Fargate [2][3].

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-task-storage.html
[2] #736
[3] #710

[manugupt1]

Hello!

To run a container an image as non-root, a customer can do the following:

They can export the path that they want to export as a VOLUME and run chown in their own Dockerfile or ImageFile.

As an example,
Let’s consider an image that has node as the base image (node will use nodejs environment) and it wants to use /var/log/exported in node:node as the user and group respectively. Now, they can
specify VOLUME directive to /var/log/exported and then all the permissions will be reflected in their task volumes.

To understand how this can be achieved, let us look at the following Dockerfile.

FROM node:12-slim ## A node js base image
RUN chown node:node /var/log/exported ## Changing permissions from root to node
VOLUME ["/var/log/exported"] ## Specifying a VOLUME directive applies the permission

Please let us know if it works for you. Thank you

[tarun-wadhwa-mmt]

RUN chown node:node /var/log/exported ## Changing permissions from root to node
VOLUME ["/var/log/exported"] ## Specifying a VOLUME directive applies the permission

This worked fine till Platform Version 1.3.0 but fails with 1.4.0
Now when 1.4.0 will be the new default, need a new workaround

[jpradelle]

I crashed a stack on that issue, something is really needed here.
As I had this issue I was not using version 1.4.0. I didn't fill PlatformVersion and got 1.3.0 previously by default, now that changed to 1.4.0, which still have this issue.

[gokhanoner]

I ended up using an EFS volume

[manugupt1]

Hello,

We have recently updated our documentation with some examples on how to use bind-mounts. These examples include:

1. Getting an empty data-volume for one or more containers.
2. To expose a path and its contents from a Dockerfile to a container.
3. To run a particular data-volume in a non-root environment.
   The link to our updated documentation can be found here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html#bind-mount-examples

If this does not solve your use-case, please reach out to AWS Support.

Thanks
Manu

[mattmassicotte]

I've looked closely at the Dockerfile my application uses, and compared it to the examples. While there are differences, as far as I can tell they are semantically equivalent, yet do not work with Fargate bind mounts. The process is ultimately unable to interact with the mounted volume.

Has anyone actually be able to make this work?

[manugupt1]

Hi Matt and Gökhan,

Can you please open a support case, so that we can work with you to understand why this does not work?

Thanks
Manu

[mattmassicotte]

@manugupt1 I'm afraid not, as technical support isn't covered by my current service level. However, I can share the Dockerfile with you, in case you want to inspect the differences.

https://github.com/apache/druid/blob/753bce324bdf8c7c5b2b602f89c720749bfa6e22/distribution/docker/Dockerfile

[manugupt1]

Hi Matt,
I looked through your Dockerfile and found that a symlink is the present for VOLUME directive.
Ref: https://github.com/apache/druid/blob/753bce324bdf8c7c5b2b602f89c720749bfa6e22/distribution/docker/Dockerfile#L38

We have updated our documentation to say that the VOLUME directive should map to an absolute path. This can be found at
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html#bind-mount-considerations

Thanks
Manu

[mattmassicotte]

@manugupt1 that was really awesome of you. Thanks so much for digging in. I'll bring this up with the dockerfile maintainers.

[jellevanhees]

To me, this is still a very valid request.

We have an ecs service based on 3 public Bitnami containers.
We want to exchange volumes between these containers one for config one for data.
Al these containers run as the same non-root user.
Yet in fargate we cant run this setup because we can not manipulate the uid/gid of the named volume.
In this case, we have no control over the docker files so we can expose a volume that way.