E_FILE_NOT_FOUND - error deleting files when using S3 Storage when endpoint points to folder

[joshualyness]

Preflight Checklist

• I have completed all Troubleshooting Steps.
• I'm on the latest version of Directus.
• There's no other issue that already describes my problem.

Describe the Bug

When deleting media from the library, I'm getting this error message when using an S3 bucket for storage:

{
  "errors": [
    {
      "message": "E_FILE_NOT_FOUND: The file ae144cff-f038-4f0a-b5b2-24dda71148bd doesn't exist\nThe specified key does not exist.",
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    }
  ]
}

I have my config set up like this:

STORAGE_LOCATIONS="s3"
STORAGE_S3_DRIVER="s3"
STORAGE_S3_KEY="----my key------"
STORAGE_S3_SECRET="-----my secret------"
STORAGE_S3_BUCKET="-----bucket name------"
STORAGE_S3_REGION="us-east-2"
STORAGE_S3_ENDPOINT="------endpoint--------"
STORAGE_S3_ACL="private"

It saves media to the bucket correctly, just wont delete it. I'm concerned that my bucket will fill up fast with old media, with no way of deleting the disused media.

The bucket has an IAM group set up with full s3 permissions for testing purposes. Any ideas if/when this can be fixed?

I'm assuming it's still trying to delete the file locally despite it being in a bucket.

To Reproduce

Set up an S3 bucket and attach a full access IAM policy.

Set up config as per documentation.

Upload media, then delete afterwards. Should present the same error message.

Errors Shown

{
  "errors": [
    {
      "message": "E_FILE_NOT_FOUND: The file ae144cff-f038-4f0a-b5b2-24dda71148bd doesn't exist\nThe specified key does not exist.",
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    }
  ]
}

What version of Directus are you using?

9.5.1

What version of Node.js are you using?

16.13.1

What database are you using?

MySQL 5.7

What browser are you using?

Chome

What operating system are you using?

macOS

How are you deploying Directus?

In an EC2 instance

[joshualyness]

I fixed the issues I was having by changing the STORAGE_S3_ENDPOINT="http://s3.us-east-2.amazonaws.com/----bucket name-----" to STORAGE_S3_ENDPOINT="http://s3.us-east-2.amazonaws.com/".

It seems that if you try to store files in a subfolder on s3, it doesn't allow you to delete files. I'll leave this open as this has to be a bug?

[rijkvanzanten]

Thanks for researching that @joshualyness! There must be a difference in either @directus/drive or the amazon s3 sdk in how it handles the endpoint with or without bucket suffix between create and delete 🤔

[rijkvanzanten]

It's been a minute, but I think this is a configuration thing after all 🤔 S3's endpoint doesn't include the bucket (that's what the BUCKET env var is for), and root folders can be set through STORAGE_S3_ROOT 🙂

Happy to keep discussing though 💪🏻