hostname underscores

[marcstreeter]

Getting Invalid endpoint error with minio services that have a hostname with underscores like:

ValueError: Invalid endpoint: http://name_with_underscore:9000

Expected Behavior

Support underscores for minio service names

Current Behavior

given .env file

ACCESS_KEY=some$random1$root2$User
SECRET_KEY=some$random1$root2$Password3
REGION=us-east-1

with a docker-compose.yml file like below:

version: "2"
volumes:
  storage_vol:
services:
  api:
    image: jdeskins/aws-python
    environment:
      AWS_ACCESS_KEY_ID: ${ACCESS_KEY}
      AWS_SECRET_ACCESS_KEY: ${SECRET_KEY}
      AWS_DEFAULT_REGION: ${REGION}
    command: python -c 'import boto3;from botocore.client import Config;boto3.resource("s3", config=Config("s3v4"), endpoint_url="http://storage:9000").buckets.all()'
  storage:
    image: minio/minio:RELEASE.2021-08-05T22-01-19Z
    volumes:
    - storage_vol:/data
    ports:
    - 9000:9000
    - 9019:9019
    environment:
      MINIO_REGION_NAME: us-east-1
      MINIO_ROOT_USER: ${ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${SECRET_KEY}
    command: server --console-address ':9019' /data

It works fine. It should work also in another (potentially larger) docker-compose.yml file like so:

version: "2"
volumes:
  storage_vol:
services:
  api:
    image: jdeskins/aws-python
    environment:
      AWS_ACCESS_KEY_ID: ${ACCESS_KEY}
      AWS_SECRET_ACCESS_KEY: ${SECRET_KEY}
      AWS_DEFAULT_REGION: ${REGION}
      # changed below from `http://storage:9000` to `http://foo_storage:9000`
    command: python -c 'import boto3;from botocore.client import Config;boto3.resource("s3", config=Config("s3v4"), endpoint_url="http://foo_storage:9000").buckets.all()'
  # changed name to include underscore, "_"
  foo_storage:
    image: minio/minio:RELEASE.2021-08-05T22-01-19Z
    volumes:
    - storage_vol:/data
    ports:
    - 9000:9000
    - 9019:9019
    environment:
      MINIO_REGION_NAME: us-east-1
      MINIO_ROOT_USER: ${ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${SECRET_KEY}
    command: server --console-address ':9019' /data

However because the service now has an underscore in the name, calls to it fail with "Invalid endpoint".

Possible Solution

update the regex, I believe a similar solution was mentioned here minio/minio-dotnet#238 and committed here https://github.com/minio/minio-dotnet/pull/334/files

Steps to Reproduce (for bugs)

run the second example I posted (with it's accompanying .env file present) and it will show the error

Context

It has made me be wary of my naming schemes used in docker-compose files. I walk through the halls, concerned that they might fail. Every corner I take, I look over my shoulder in suspicion. Life has become definitely harder.

Regression

No

Your Environment

• Version used (minio --version): minio/minio:RELEASE.2021-08-05T22-01-19Z
• Server setup and configuration: docker-compose
• Operating System and version (uname -a): minio docker image (redhat?) and python using boto to connect

[harshavardhana]

Getting Invalid endpoint error with minio services that have a hostname with underscores like:

ValueError: Invalid endpoint: http://name_with_underscore:9000

Expected Behavior

Support underscores for minio service names

Current Behavior

given .env file

ACCESS_KEY=some$random1$root2$User
SECRET_KEY=some$random1$root2$Password3
REGION=us-east-1

with a docker-compose.yml file like below:

version: "2"
volumes:
  storage_vol:
services:
  api:
    image: jdeskins/aws-python
    environment:
      AWS_ACCESS_KEY_ID: ${ACCESS_KEY}
      AWS_SECRET_ACCESS_KEY: ${SECRET_KEY}
      AWS_DEFAULT_REGION: ${REGION}
    command: python -c 'import boto3;from botocore.client import Config;boto3.resource("s3", config=Config("s3v4"), endpoint_url="http://storage:9000").buckets.all()'
  storage:
    image: minio/minio:RELEASE.2021-08-05T22-01-19Z
    volumes:
    - storage_vol:/data
    ports:
    - 9000:9000
    - 9019:9019
    environment:
      MINIO_REGION_NAME: us-east-1
      MINIO_ROOT_USER: ${ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${SECRET_KEY}
    command: server --console-address ':9019' /data

It works fine. It should work also in another (potentially larger) docker-compose.yml file like so:

version: "2"
volumes:
  storage_vol:
services:
  api:
    image: jdeskins/aws-python
    environment:
      AWS_ACCESS_KEY_ID: ${ACCESS_KEY}
      AWS_SECRET_ACCESS_KEY: ${SECRET_KEY}
      AWS_DEFAULT_REGION: ${REGION}
      # changed below from `http://storage:9000` to `http://foo_storage:9000`
    command: python -c 'import boto3;from botocore.client import Config;boto3.resource("s3", config=Config("s3v4"), endpoint_url="http://foo_storage:9000").buckets.all()'
  # changed name to include underscore, "_"
  foo_storage:
    image: minio/minio:RELEASE.2021-08-05T22-01-19Z
    volumes:
    - storage_vol:/data
    ports:
    - 9000:9000
    - 9019:9019
    environment:
      MINIO_REGION_NAME: us-east-1
      MINIO_ROOT_USER: ${ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${SECRET_KEY}
    command: server --console-address ':9019' /data

However because the service now has an underscore in the name, calls to it fail with "Invalid endpoint".

Possible Solution

update the regex, I believe a similar solution was mentioned here minio/minio-dotnet#238 and committed here https://github.com/minio/minio-dotnet/pull/334/files

Steps to Reproduce (for bugs)

run the second example I posted (with it's accompanying .env file present) and it will show the error

Context

It has made me be wary of my naming schemes used in docker-compose files. I walk through the halls, concerned that they might fail. Every corner I take, I look over my shoulder in suspicion. Life has become definitely harder.

Regression

No

Your Environment

• Version used (minio --version): minio/minio:RELEASE.2021-08-05T22-01-19Z
• Server setup and configuration: docker-compose
• Operating System and version (uname -a): minio docker image (redhat?) and python using boto to connect

Invalid endpoint is returned by boto3 here why is it reported here ?

[harshavardhana]

The fix needs to be in boto3 client here, which does not support underscore DNS names.

Note hostnames cannot have underscore as per the RFC

The underscore is not a legal character for use in hostnames. As defined in RFC 822, the only legal characters are the following:

• Alphanumeric (a-z and 0-9): Both uppercase and lowercase letters are acceptable, and the hostname is case insensitive. In other words, dvader.empire.gov is identical to DVADER.EMPIRE.GOV and Dvader.Empire.Gov.
• Hyphen (-): Neither the first nor the last character in a hostname field should be a hyphen.
• Period (.): The period should be used only to delimit fields in a hostname (e.g., dvader . empire . gov) .

[marcstreeter]

Yep, yep, thank you @harshavardhana, definitely boto boto/boto3#703