Override service endpoint URL through configuration files/environment variables

[kellertk]

Summary
It would be nice to be able to override a service endpoint URL through configuration files, environment variables, or some other runtime mechanism instead of only in code.

Description
This issue de-duplicates several issues and pull requests from other AWS SDKs and various tools, specifically:

• AWS CLI GitHub issue Ability to specify endpoint-url in profile aws-cli#1270
• AWS CLI GitHub issue Provide environment variables to override default endpoint URL aws-cli#4454
• AWS CLI GitHub issue Ability to pass multiple service endpoints aws-cli#4012
• Python SDK boto3 GitHub issue Provide environment variables to override default endpoint URL boto/boto3#2099
• Python SDK boto3 GitHub PR Add environment variable to override endpoint_url (#2099) boto/boto3#2746
• AWS PHP SDK issue Feature: Configure Services' Endpoint URLs via Environment Variables aws-sdk-php#1239
• AWS JavaScript SDK issue Feature: Configure Services' Endpoint URLs via Environment Variables aws-sdk-js#3236
• AWS JavaScript SDK issue Add environment variable to override endpoint_url aws-sdk-js#3999

This Stack Overflow post also describes the feature request: https://stackoverflow.com/questions/32618216/override-s3-endpoint-using-boto3-configuration-file.

First of all, we’d like to thank everyone for their feedback and patience while we examine these requests and determine how to move forward. Up front, there is an assumption in some of the threads linked above that AWS doesn’t want to implement this because it encourages reuse of the SDK for non-AWS products:

Is it possible that this PR is not merged just because it allows developers to use other S3-like storage?
In that case, there is strong a conflict of interest between AWS services and the community. If I am right, this PR will never be merged in foreseeable future.

This is not the case, and we want to be clear that it is not a conflict of interest preventing the merging of that specific pull request. As @CodePint mentioned, “The functionality is already already exposed via the endpoint_url parameter which can be passed to the boto3 client function, the ability to configure it using an env var is purely an extension of that.” In the linked Stack Overflow post, a few commenters posted the same thing. When instantiating a service client in a recent-ish version of the SDK, you can specify the endpoint URL for that service. This feature request doesn’t change that functionality, but instead extends it through the use of a configuration file or environment variable.

So again, today everyone should be able to specify an endpoint URL for a service, but it needs to be done in the service client instantiation. This feature request is to explore methods that might make this more convenient or more ergonomic.

Implementation Challenges
A handful of issues have been opened across AWS SDKs requesting the ability to customize an endpoint URL either via an environment variable or profile configuration. There are many trade-offs to consider here that make the implementation of this change more challenging than it might seem. For example, a maintainer described in this comment how:

The thing with --endpoint-url is that the value set will most likely not apply across differing services. So if you configure a global endpoint url, it may work for one service, but you will run into errors if you try to make requests to other services.

This implementation detail is one item we need to consider (and configuration file management in general), but there are others. Before moving forward, we must consider how this feature will work across SDKs to ensure a consistent experience and maintain backwards compatibility. One important point is that we don’t want to implement this in only one of the SDKs, or only in the AWS CLI. In general, all of the AWS tools follow the same configuration file/environment variable syntax and we want to keep it that way - as @kapilt suggested, this change would need to be coordinated across the SDKs for consistency.

Next Steps
We acknowledge the demand for this feature and are actively discussing the options internally. Expect an update on this thread within a week with our progress on our decisions here. I can promise that we won’t have this done in a week - but I can also promise that by that time we’ll have had our internal discussion and be able to present some options for further discussion with the community. Anything that I can share, I will here.

We use 👍 to track community engagement on issues, so if this issue is relevant to you, please react with a 👍 emoji. Some of the linked threads above have a substantial number of reactions, which does help us gauge this feature’s important to our customers. If you’d like, you can also post more specific feedback relating to how you’d use a configuration file/environment managed endpoints and why the current method of specifying this info in the service client instantiation doesn’t work for you. We’ll take that into consideration as we design the feature.

[flickerfly]

those guys who were interested in the previous issue are dead now

"I'm not dead yet." - Monty Python

[codyhazelwood]

I would be happy to see this implemented as environment variables. We currently do all of our AWS configuration with environment variables in development. Being able to set the endpoint there would allow us to call Lambdas in local SAM and S3 buckets with local MinIO, while reverting to the official AWS endpoints in prod with no code changes.

Of course, we can use a wrapper around the specific SDKs, but that isn't as nice as being able to directly import the SDK without thinking about it.

[kdaily]

Hi all,

Thanks for your patience. I just linked a pull request (#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file.

You can skip directly to read the proposal here.

🗒️ Providing feedback

If you're already here and have high level feedback, we'll read and review and comments directly on this issue. If you have feedback on specific parts of the proposal, we'd appreciate if you provide feedback via the GitHub pull request by commenting on the lines where you have questions or suggestions.

After reading the proposal and comments, you can give feedback on the pull request in the following ways:

1. If you generally approve of the direction, add 👍 reaction to top-level comment.
2. If you prefer one of the alternatives listed, comment on the line where an alternative is proposed that you prefer. If there is already a comment for that alternative, add a 👍 to that comment.
3. If there are any other suggestions/alternatives that you'd prefer, add a comment describing the suggestion and alternative. 👍 the comment if you agree with it.
4. Comment with any additional questions.

Thank you!