Allow underscores in endpoint_url hostname

[rymir]

Currently is_valid_endpoint in botocore.utils doesn't allow for underscores in the endpoint_url.

We weren't able to specify docker containers names using underscores as endpoints, so we wrote the following fix.

It also adds an explicit call to bool() in the is_valid_endpoint return statement to conform to the function contract stating that it returns True or False.

[codecov-io]

Current coverage is 97.31%

Merging #960 into develop will not change coverage

@@            develop       #960   diff @@
==========================================
  Files            43         43          
  Lines          7014       7014          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits           6826       6826          
  Misses          188        188          
  Partials          0          0          

Powered by Codecov. Last updated by fac5529...7c587a8

[jamesls]

Thanks for the pull request. I'm hesitant to change this, because I'd prefer to stick to what the RFC says to use for valid hostnames. There's a good summary here:

The Internet standards (Requests for Comments) for protocols mandate that component hostname labels may contain only the ASCII letters 'a' through 'z' (in a case-insensitive manner), the digits '0' through '9', and the hyphen ('-'). The original specification of hostnames in RFC 952, mandated that labels could not start with a digit or with a hyphen, and must not end with a hyphen. However, a subsequent specification (RFC 1123) permitted hostname labels to start with digits. No other symbols, punctuation characters, or white space are permitted.

cc @kyleknap @JordonPhillips

[kyleknap]

Yeah I would prefer to stick to the RFC, but I think this is validation is something that we should allow people to opt out of. Could we make the PR do that instead? Maybe make the validation of endpoint url a handler to unregister or part of the param_validation config option?

[jamesls]

I'd prefer not to complicate the code unless this is a reasonably common use case.

Can we get an example of how docker containers and botocore's endpoint_url are being used? It's not clear to me why you'd be setting botocore's endpoint_url to a docker container name.

[rymir]

Right, I didn't know about https://tools.ietf.org/html/rfc952.
@jamesls, when using docker-compose, to connect to one container from an other one you can use the service container name. In our case we used underscores in the name (ex_dynamo), so the resulting endpoint was http://ex_dynamo:8000, but given the RFC guidelines you guys pointed out, we can easily stop using underscores.

Thanks, I'm closing this one.

[remram44]

Everybody using Docker-Compose is going to run into this. You are the only HTTP client out there that refuses those URLs. Please don't do insane, unusual sanity checks that no one else does, don't provide any additional safety and breaks actual code.

[inieves]

@jamesls Earlier in this thread you asked about a use case where botocore's endpoint_url will be set to a docker container name. Here is such a use case:

Developer is building a development environment (or even a production environment) and wants to interface with AWS-compatible (but not Amazon AWS) services via AWS CLI... and in fact those AWS-compatible services are a part of the docker deployment. In this case, the developer will name the services/hosts/endpoints x_y_z... and will try to pass those to AWS CLI... which will then fail because botocore does not like underscore (or some other botocore restriction)

A use case that will probably become more common is testing and using self-hosted AWS-compatible services within the local deployment... and thus using endpoints that are not necessarily correct in the eyes of some RFC.

[and3rson]

Same issue here. We have a lambda named some_lambda_name and want to use the same name for Docker container with lambci image. But now we have to name it somelabdaname to make botocore accept the URL that looks like http://somelambdaname:9001. This breaks consistency in the code since we need to have a lot of IFs for such cases as well as we need to use confusing names for those hostnames:

if dev_env:
    client = boto3.Client('lambda', endpoint_url='http://somelambdaname:9091')
else:
    client = boto3.Client('lambda')
client.invoke(FunctionName='some_lambda_name')  # Cannot reuse the hostname without any extra magic.

Now when somebody walks along and asks: "why do we strip underscores in endpoint_url?" - I can point them at this conversation.

TL;DR: Presence or absence of underscores in the URL should be a decision of the developer, NOT the decision of the botocore. This validation is some totally unnecessary paranoid action from botocore. We want to decide ourselves if we want to follow the RFC and use underscores or not.

Relates to boto/boto3#703, #1042 & #1013.