[Bug]: AWS S3 bucket access denied

[dhanush-2501]

dstack version

0.10.4

Python version

3.10.9

Host OS

macOS 13.4.1

Host Arch

ARM

What happened?

While setting up the project in the dstack hub,after entering AWS credentials no regions and buckets got listed.dstack start command resulted in botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Relevant log output

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/dhanush/Library/Python/3.10/lib/python/site-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "/Users/dhanush/Library/Python/3.10/lib/python/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/base.py", line 78, in call_next
    message = await recv_stream.receive()
  File "/Users/dhanush/Library/Python/3.10/lib/python/site-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/homebrew/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/opt/homebrew/lib/python3.10/site-packages/dstack/_internal/hub/main.py", line 77, in app_logging
    response = await call_next(request)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/homebrew/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/homebrew/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/homebrew/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/homebrew/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/opt/homebrew/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/opt/homebrew/lib/python3.10/site-packages/dstack/_internal/hub/routers/projects.py", line 36, in get_backend_config_values
    result = await run_async(configurator.configure_project, config.__root__)
  File "/opt/homebrew/lib/python3.10/site-packages/dstack/_internal/hub/utils/common.py", line 5, in run_async
    return await asyncio.get_running_loop().run_in_executor(None, func, *args)
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/dstack/_internal/hub/services/backends/aws/configurator.py", line 82, in configure_project
    project_values.s3_bucket_name = self._get_hub_buckets_element(
  File "/opt/homebrew/lib/python3.10/site-packages/dstack/_internal/hub/services/backends/aws/configurator.py", line 181, in _get_hub_buckets_element
    response = s3_client.list_buckets()
  File "/opt/homebrew/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/botocore/client.py", line 964, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Additional Information

Steps to reproduce

• dstack start
• head over to the dstack hub and click add
• choose AWS as backend type
• enter access ID
• enter Secret access key
• check the dstack start command terminal output

Policy used

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetLifecycleConfiguration",
                "s3:PutLifecycleConfiguration",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:DeleteObjectTagging",
                "s3:GetBucketAcl"
            ],
            "Resource": [
                "arn:aws:s3:::dstack-test-bucket",
                "arn:aws:s3:::dstack-test-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:FilterLogEvents",
                "logs:TagLogGroup",
                "logs:CreateLogGroup",
                "logs:CreateLogStream"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/dstack/jobs/dstack-test-bucket*:*",
                "arn:aws:logs:*:*:log-group:/dstack/runners/dstack-test-bucket*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:UpdateSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:CreateSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:PutResourcePolicy",
                "secretsmanager:TagResource",
                "secretsmanager:DeleteSecret"
            ],
            "Resource": [
                "arn:aws:secretsmanager:*:*:secret:/dstack/dstack-test-bucket/credentials/*",
                "arn:aws:secretsmanager:*:*:secret:/dstack/dstack-test-bucket/secrets/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:RunInstances",
                "ec2:CreateTags",
                "ec2:CreateSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CancelSpotInstanceRequests",
                "ec2:TerminateInstances"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/dstack_bucket": "dstack-test-bucket"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:TagRole"
            ],
            "Resource": "arn:aws:iam::*:role/dstack_role_dstack_test_bucket*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:TagPolicy"
            ],
            "Resource": "arn:aws:iam::*:policy/dstack_policy_dstack_test_bucket*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetInstanceProfile",
                "iam:CreateInstanceProfile",
                "iam:AddRoleToInstanceProfile",
                "iam:TagInstanceProfile",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::*:instance-profile/dstack_role_dstack_test_bucket*",
                "arn:aws:iam::*:role/dstack_role_dstack_test_bucket*"
            ]
        }
    ]
}