Deleting an external storage file

[arladmin]

Preflight Checklist

• I have completed all Troubleshooting Steps.
• I'm on the latest version of Directus.
• There's no other issue that already describes my problem.

Describe the Bug

I have an external (S3) storage configured.
When delete a file (directus_files) from the Admin app/API, i expected the file to also get deleted from the external storage, just like it is for local files. But that didn't happen.

This would create a huge issue where there could potentially be thousands of orphan files left in the external storage bucket/folder attached to Directus.

To Reproduce

1. Attach an external S3 storage to Directus
2. Upload a file to the said storage
3. Now delete it. The file gets deleted from Directus but is still found in the external storage

What version of Directus are you using?

v9.0.0-rc.92

What version of Node.js are you using?

16.5.0

What database are you using?

Postgres 12

What browser are you using?

Chrome

What operating system are you using?

Ubuntu

How are you deploying Directus?

Docker

[emrahnazif]

I confirm same issue.
latest version of directus with digitalocean s3 storage.

To Reproduce:

• clean install directus
• setup a s3 storage
• upload a file
• delete it
• check the s3 storage

What version of Directus are you using?
v9.0.0-rc.92
What version of Node.js are you using?
16.5.0
What database are you using?
Postgres 12
What browser are you using?
Chrome
What operating system are you using?
Ubuntu 18.04
How are you deploying Directus?
digitalocean marketplace

[paulboudewijn]

Perhaps this is related to issue #8899 and PR #9126.

[marvinisaac]

I'm getting the same issue while also using DigitalOcean's Spaces storage.

To reproduce:

• Run Directus via docker-compose
• Setup and connect DO Spaces
• Upload an image (Success)
• Delete (Error message in UI and in Docker logs attached)
• Database record gets deleted
• Image is still in storage (Filename is same as in error but with .jpg extension)

What version of Directus are you using?
v9.0.1

What version of Node.js are you using?
n/a (Comes with the container image)

What database are you using?
MariaDB 10.6.4

What browser are you using?
Chrome

What operating system are you using?
Windows 10

How are you deploying Directus?
Docker / docker-compose

[emrahnazif]

This is still a problem for me. Any solution to this?
v9.4.2

[marvinisaac]

I switched from DigitalOcean Spaces to Google Cloud Storage. Everything is working OK now

[calebyusuf]

This is still a problem for me. Any solution to this? v9.4.2
Although I am running v9.5.1

I was still getting this issue but just resolved it by adding the ListBucket and ListBucketVersions permissions to my S3 bucket policy (although i don't think ListBucketVersions is necessary)

List Bucket is used in node_modules/@directus/drive-s3/dist/AmazonWebServicesS3Storage.js

Hope this helps!!

[rijkvanzanten]

I think there has been an update in this cold-case: #11641

Could you confirm the value used for ENDPOINT @calebyusuf?

[tarcon]

This still does not seem to work in 9.8 using S3 as storage adapter

Deleting via the admin app or DELETE /files/:id rest api is successful (204) and removes the metadata in directus_files, but leaves the files in S3

[rijkvanzanten]

@tarcon What AWS permissions are you using for the S3 bucket for the credentials used with Directus?

[tarcon]

The directus log with loglevel trace does not show an error while deleting files.

My private bucket has disabled ACLs. I have not set up iam policies for the access.
I work with a pretty lightweight config. The directus server accesses the bucket directly within a secure private network.

Files work fine, except for the deletion.

Am I missing something?

STORAGE_LOCATIONS: "s3",
STORAGE_S3_DRIVER: "s3",
STORAGE_S3_ROOT: "./files",
STORAGE_S3_KEY: "",
STORAGE_S3_SECRET: "",
STORAGE_S3_BUCKET: bucket.bucketName,
STORAGE_S3_REGION: "eu-central-1",
STORAGE_S3_ENDPOINT: "s3.amazonaws.com",
STORAGE_S3_ACL: "",

[rijkvanzanten]

I'm assuming it's because the bucket might be configured to allow anybody to put files, but not delete files 🤔

[rijkvanzanten]

We haven't been able to reproduce this so far.. For all the instances we've debugged, it has been due to access control rules in S3 itself (eg incorrect configuration for s3:deleteObject and/or other S3 rules). I'll close this for now, but happy to keep discussing / debugging 👍🏻

[mattbisme]

I believe that I have figured this out. I've been using Backblaze's B2 (S3 compatible) storage; I was getting the same problem. The short answer seems to be that S3 storage does not like relative paths.

So, in @tarcon's example, change

STORAGE_S3_ROOT="./files"

to

STORAGE_S3_ROOT="/files"

This is what worked for me, at least.

It might be worth updating the docs to reflect this. I also had the immediate thought that it was a permissions issue of some kind, but it is so much simpler than that!

[rijkvanzanten]

cc @erondpowell

[tarcon]

I can confirm this fixes my problem. Good job!

[chris-mcdonald-dev]

Sorry for bringing attention to this again. I'm having the same issue as @tarcon. So, no error when deleting through Directus but it's not actually deleting any files in S3. But my STORAGE_ROOT config already does not use a relative path. I assumed this was a permissions issue as well, but my S3 policy also includes s3:deleteObject specifically only for the root folder, and I'm using the key/secret from an IAM user with that policy assigned.

I've confirmed the permissions allow for deletion of the image files with AWS's SDK.

Directus .env

STORAGE_LOCATIONS="aws"

## AWS S3
STORAGE_AWS_ROOT="/folder"
STORAGE_AWS_DRIVER="s3"
STORAGE_AWS_BUCKET="bucket-name"
STORAGE_AWS_REGION="us-east-1"
STORAGE_AWS_ENDPOINT="s3.amazonaws.com"
STORAGE_AWS_KEY="iam-user-key"
STORAGE_AWS_SECRET="iam-user-secret"

AWS Policy Config

{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:GetObject",
    "s3:DeleteObject"
  ],
  "Resource": "arn:aws:s3:::bucket-name/folder/*"
}

Any idea what I might be doing wrong?

[rijkvanzanten]

@chris-mcdonald-dev Mind opening a new issue? This one was marked resolved nearly exactly a year ago, which means that your problem is very likely caused by something else 🙂