Allow multiple volumes with managed EFS for a single service

[efekarakus]

We should enable more than one storage.volumes with managed EFS for a single service.

Ask

Original ask: #1559 (comment)

Today users are limited to a single EFS access point per service:

storage:
  volumes:
    frontend:
      efs:
        uid: 123
        gid: 123
      path: /var/data

If they try to use the same EFS filesystem for multiple volumes like this:

storage:
  volumes:
    v1:
      efs:
        uid: 123
        gid: 123
      path: /etc/mount1
    v2:
      efs:
        uid: 456
        gid: 456
      path: /etc/mount2

We return the following error:

validate "storage": cannot specify more than one managed volume per service

Programming model

1. In order to support multiple access points for a single service we need to transform the following resource into a list:
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml

   Lines 2 to 3 in 2cc2403

   	{{- if .Storage.ManagedVolumeInfo}}
   	AccessPoint:

2. We need to assign a new RootDirectory.Path for the additional access points. Today, we hard-code it to the service name like "frontend". The proposal is to do <svcName>-<volumeName>.
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml

   Lines 13 to 14 in 2cc2403

   	RootDirectory:
   	Path: !Sub '/{{.Storage.ManagedVolumeInfo.DirName}}'

3. Update the IAM policy in the TaskRole to grant access to multiple access points:
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/taskrole.yml

   Lines 72 to 74 in 2cc2403

   	{{- if .Storage.ManagedVolumeInfo}}
   	- PolicyName: 'GrantAccessCopilotManagedEFS'
   	PolicyDocument:

[FraserThompson]

Just in case this helps prioritize this... Multiple access points is essential for running CMSs like Drupal (probably others) which save user files in multiple locations. For example:

Public filesystem in /opt/drupal/web/sites/default/files
Private fileystem /opt/drupal/private

A single access point couldn't meet this use case. It's possible with externally managed filesystems, but having all this managed by Copilot would be much more convenient.

[efekarakus]

Hi @FraserThompson ! out of curiosity wouldn't this scenario be satisfied with:

storage:
  volumes:
    frontend:
      efs: true
      path: /opt/drupal

Or am I misunderstanding a step?

[FraserThompson]

Hi @efekarakus ! This would work to persist files, yes, but as far as I can tell it would go against best practices.

The other contents of /opt/drupal come from the Docker image and should only exist in ephemeral storage because they're not stateful. The only stateful data is contained in those two directories.

Ideally you could put all files you want persisted into one directory and make that an EFS volume, but Drupal doesn't offer full control over where these files can be.

Also there are other factors, like we might want to share just the public files with an NGINX sidecar so it can serve static assets. We might have another sidecar running Solr for search, and this will also need to persist the search index somewhere. I don't think it would make sense for all of these to persist their data to the same place.

For reference the AWS example for Drupal on ECS uses one filesystem with multiple access points for each directory they want to persist. Unless I'm misunderstanding something (which is entirely possible) I don't think we can do this right now with Copilot.

[hardeepjethwani]

We should enable more than one storage.volumes with managed EFS for a single service.

Ask

Original ask: #1559 (comment)

Today users are limited to a single EFS access point per service:

storage:
  volumes:
    frontend:
      efs:
        uid: 123
        gid: 123
      path: /var/data

If they try to use the same EFS filesystem for multiple volumes like this:

storage:
  volumes:
    v1:
      efs:
        uid: 123
        gid: 123
      path: /etc/mount1
    v2:
      efs:
        uid: 456
        gid: 456
      path: /etc/mount2

We return the following error:

validate "storage": cannot specify more than one managed volume per service

Programming model

1. In order to support multiple access points for a single service we need to transform the following resource into a list:
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml

   Lines 2 to 3 in 2cc2403

   	{{- if .Storage.ManagedVolumeInfo}}
   	AccessPoint:

2. We need to assign a new RootDirectory.Path for the additional access points. Today, we hard-code it to the service name like "frontend". The proposal is to do <svcName>-<volumeName>.
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml

   Lines 13 to 14 in 2cc2403

   	RootDirectory:
   	Path: !Sub '/{{.Storage.ManagedVolumeInfo.DirName}}'

3. Update the IAM policy in the TaskRole to grant access to multiple access points:
   

   copilot-cli/internal/pkg/template/templates/workloads/partials/cf/taskrole.yml

   Lines 72 to 74 in 2cc2403

   	{{- if .Storage.ManagedVolumeInfo}}
   	- PolicyName: 'GrantAccessCopilotManagedEFS'
   	PolicyDocument:

Hi @efekarakus , Do we have any update on this issue resolution?