S3 putObject fails when starting up project: What permissions are required.?

[sudhakartag1]

What happened?

I'm trying to run a project created with cookiecutter using docker compose up -f production.yaml. On an EC2 instance with an instance role attached with s3:* on the S3 bucket configured with DJANGO_AWS_STORAGE_BUCKET_NAME. However the app fails with:

django-1        | Traceback (most recent call last):
django-1        |   File "/app/manage.py", line 32, in <module>
django-1        |     execute_from_command_line(sys.argv)
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
django-1        |     utility.execute()
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
django-1        |     self.fetch_command(subcommand).run_from_argv(self.argv)
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 412, in run_from_argv
django-1        |     self.execute(*args, **cmd_options)
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 458, in execute
django-1        |     output = self.handle(*args, **options)
django-1        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/collectfast/management/commands/collectstatic.py", line 91, in handle
django-1        |     ret = super().handle(**options)
django-1        |           ^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
django-1        |     collected = self.collect()
django-1        |                 ^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/collectfast/management/commands/collectstatic.py", line 71, in collect
django-1        |     return super().collect()
django-1        |            ^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
django-1        |     handler(path, prefixed_path, storage)
django-1        |   File "/usr/local/lib/python3.12/site-packages/collectfast/management/commands/collectstatic.py", line 133, in copy_file
django-1        |     self.maybe_copy_file(args)
django-1        |   File "/usr/local/lib/python3.12/site-packages/collectfast/management/commands/collectstatic.py", line 117, in maybe_copy_file
django-1        |     super().copy_file(path, prefixed_path, source_storage)
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 378, in copy_file
django-1        |     self.storage.save(prefixed_path, source_file)
django-1        |   File "/usr/local/lib/python3.12/site-packages/django/core/files/storage/base.py", line 38, in save
django-1        |     name = self._save(name, content)
django-1        |            ^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/storages/backends/s3.py", line 494, in _save
django-1        |     obj.upload_fileobj(content, ExtraArgs=params, Config=self.transfer_config)
django-1        |   File "/usr/local/lib/python3.12/site-packages/boto3/s3/inject.py", line 731, in object_upload_fileobj
django-1        |     return self.meta.client.upload_fileobj(
django-1        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/boto3/s3/inject.py", line 642, in upload_fileobj
django-1        |     return future.result()
django-1        |            ^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/s3transfer/futures.py", line 103, in result
django-1        |     return self._coordinator.result()
django-1        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/s3transfer/futures.py", line 266, in result
django-1        |     raise self._exception
django-1        |   File "/usr/local/lib/python3.12/site-packages/s3transfer/tasks.py", line 139, in __call__
django-1        |     return self._execute_main(kwargs)
django-1        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/s3transfer/tasks.py", line 162, in _execute_main
django-1        |     return_value = self._main(**kwargs)
django-1        |                    ^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/s3transfer/upload.py", line 764, in _main
django-1        |     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
django-1        |   File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 553, in _api_call
django-1        |     return self._make_api_call(operation_name, kwargs)
django-1        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-1        |   File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
django-1        |     raise error_class(parsed_response, operation_name)
django-1        | botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

What should've happened instead?

The PutObject should have worked

Additional details

• Host system configuration:

   Amazon Linux release 2023.3.20240304 (Amazon Linux)
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.3.20240304"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"
Amazon Linux release 2023.3.20240304 (Amazon Linux)

• Python version, run python3 -V:Python 3.12.2
• Docker version (if using Docker), run docker --version: Docker version 25.0.3, build 4debf41
• docker compose version (if using Docker), run docker compose --version: Docker Compose version v2.24.6
• ...

What I've tried

After logging into the EC2 instance I can do everything on the bucket via the AWS CLI (cp, rm etc), through the AWS SDK using a simple Python program that uploads a file, from a container on the EC2 instance -including the django container. I've checked that the django container is assuming the correct role and I've also tried with a test user with the same permissions and set the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. Also works with those creds from my local machine.
There are no bucket policies on the bucket, ACLs enabled, and the role has AmazonS3FullAccess. I've also tried with a user with the same permissions and putting the credentials in the env.

https://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html#optional-use-aws-iam-role-for-ec2-instance says:
"create an IAM role and attach it to the existing EC2 instance or create a new EC2 instance with that role. The role should assume, at minimum, the AmazonS3FullAccess permission."

Are there any other permissions required, what can I check?
Thanks

[browniebroke]

The error seems to be happening during collectstatic command and the project uses collectfast, which is unmaintained (#4523). I'm wondering if the issue goes away if you remove it from INSTALLED_APPS?

Failing that, it might be worth searching for a similar issue in the django-storages repo.

[sudhakartag1]

@browniebroke awesome - that worked!
Thank you

[browniebroke]

What worked? The collectfast one or the other one?