Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

endpoint_url global option from AWS config is not working #57

Closed
IIPOCTAK opened this issue Mar 13, 2025 · 4 comments · Fixed by #58
Closed

endpoint_url global option from AWS config is not working #57

IIPOCTAK opened this issue Mar 13, 2025 · 4 comments · Fixed by #58
Labels
bug Something isn't working

Comments

@IIPOCTAK
Copy link

IIPOCTAK commented Mar 13, 2025 •
edited
Loading

Hello! Thank you for the nice S3 utility.

I found a bug with AWS config and endpoint_url option. If you set endpoint_url with DNS records and port, for example:
~/.aws/config

[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4

Then set credentials in correspond file:
~/.aws/credentials

[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin

You'll see buckets, but cannot open them and get an error:

Image
Image

In error.log I see:

2025-03-13 12:44:34.360836 +03:00 Failed to load objects: ServiceError(ServiceError { source: Unhandled(Unhandled { source: XmlDecodeError { kind: Custom("encountered invalid XML root: expected ListBucketResult but got StartEl { name: Name { prefix: \"\", local: \"ListAllMyBucketsResult\" }, attributes: [Attr { name: Name { prefix: \"\", local: \"xmlns\" }, value: \"http://s3.amazonaws.com/doc/2006-03-01/\" }], closed: false, depth: 0 }. This is likely a bug in the SDK.") }, meta: ErrorMetadata { code: None, message: None, extras: None } }), raw: Response { status: StatusCode(200), headers: Headers { headers: {"accept-ranges": HeaderValue { _private: H0("bytes") }, "content-length": HeaderValue { _private: H0("465") }, "content-type": HeaderValue { _private: H0("application/xml") }, "server": HeaderValue { _private: H0("MinIO") }, "strict-transport-security": HeaderValue { _private: H0("max-age=31536000; includeSubDomains") }, "vary": HeaderValue { _private: H0("Origin") }, "vary": HeaderValue { _private: H0("Accept-Encoding") }, "x-amz-id-2": HeaderValue { _private: H0("dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8") }, "x-amz-request-id": HeaderValue { _private: H0("182C53996FBE08C5") }, "x-content-type-options": HeaderValue { _private: H0("nosniff") }, "x-ratelimit-limit": HeaderValue { _private: H0("1045") }, "x-ratelimit-remaining": HeaderValue { _private: H0("1045") }, "x-xss-protection": HeaderValue { _private: H0("1; mode=block") }, "date": HeaderValue { _private: H0("Thu, 13 Mar 2025 09:44:34 GMT") }} }, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><Buckets><Bucket><Name>test-bucket</Name><CreationDate>2025-03-12T11:33:23.176Z</CreationDate></Bucket><Bucket><Name>test-bucket-2</Name><CreationDate>2025-03-12T11:35:32.468Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>")), retryable: true }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } })

Looks like endpoint_url is ignored and stu is using public AWS instance: s3.amazonaws.com.

If I run stu with --endpoint-url argument, everything works as expected (I see buckets and bucket's data):
stu --endpoint-url http://localhost:9000

Image
Image

How to reproduce:

I've created docker-compose.yaml file to setup MinIO (we are using private MinIO in our project), create buckets and upload test file:

---
version: "3.8"

services:
  minio:
    image: minio/minio
    container_name: minio
    restart: unless-stopped
    ports:
      - "9000:9000" 
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
      MINIO_DEFAULT_BUCKETS: test-bucket
    command: server /data --console-address ":9001"
    volumes:
      - minio_data:/data

  create-bucket:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      sleep 2;
      /usr/bin/mc alias set local http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc mb local/test-bucket 2>/dev/null;
      /usr/bin/mc mb local/test-bucket-2 2>/dev/null;
      echo 'Test file' > /tmp/test.txt;
      /usr/bin/mc cp /tmp/test.txt local/test-bucket/test.txt;
      /usr/bin/mc cp /tmp/test.txt local/test-bucket-2/test.txt;
      exit 0;
      "

volumes:
  minio_data:

Run:
docker-compose up

Create aws directory with necessary config files:

mkdir -p ~/.aws

cat << EOF > ~/.aws/config
[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4
EOF

cat << EOF > ~/.aws/credentials
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
EOF

Run stu without arguments and with --endpoint-url at the next run.

Workaround

I've added alias in my profile:
alias stu='stu --endpoint-url http://localhost:9000'
@lusingander lusingander mentioned this issue Mar 13, 2025
Merged
@lusingander
Copy link
Owner

@IIPOCTAK Thanks for reporting this!
I think it will work if you explicitly specify --path-style always.
I fixed it so that this value is automatically set if localhost is specified as endpoint_url in profile. #58

@IIPOCTAK
Copy link
Author

IIPOCTAK commented Mar 13, 2025 •
edited
Loading

@lusingander thanks for the quick fix. Will it work for non localhost endpoint_url ? For example: https://myminio.example.com:9000 ?
With --path-style always works well. Is it possible to override default value from auto to always somewhere in config?

@lusingander
Copy link
Owner

If an endpoint url is specified, it will always be treated as if --path-style always was specified.
Currently, it cannot be set in the config, but it might be a good idea to make it configurable.

@IIPOCTAK
Copy link
Author

Thank you so much! Waiting for the next release.

@lusingander lusingander mentioned this issue Mar 16, 2025
Open
@lusingander lusingander added the bug Something isn't working label Mar 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix force_path_style setting to be set correctly lusingander/stu
2 participants
@lusingander @IIPOCTAK