New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to specify endpoint-url in profile #1270

Open
StFS opened this Issue Apr 7, 2015 · 48 comments

Comments

Projects
None yet
@StFS

StFS commented Apr 7, 2015

Currently I don't seem to be able to specify an endpoint URL in my profile. I always have to specify the --endpoint-url option on the command line.

I would like to be able to do something like the following (in my ~/.aws/config file):

[default]
region = us-east-1
output = json

[profile nextcode]
region = myregion-1a
output = json
endpoint-url = http://c.my.other.aws.compatable.service.com
@kyleknap

This comment has been minimized.

Show comment
Hide comment
@kyleknap

kyleknap Apr 7, 2015

Member

@StFS
If you do not mind me asking, what is the specific use case do you need it for (like which service or services would you plan to use it for)? The thing with --endpoint-url is that the value set will most likely not apply across differing services. So if you configure a global endpoint url, it may work for one service, but you will run into errors if you try to make requests to other services.

I could see having a per-service configuration for endpoint-url, but I am having trouble imagining a global endpoint-url across all AWS services.

Member

kyleknap commented Apr 7, 2015

@StFS
If you do not mind me asking, what is the specific use case do you need it for (like which service or services would you plan to use it for)? The thing with --endpoint-url is that the value set will most likely not apply across differing services. So if you configure a global endpoint url, it may work for one service, but you will run into errors if you try to make requests to other services.

I could see having a per-service configuration for endpoint-url, but I am having trouble imagining a global endpoint-url across all AWS services.

@StFS

This comment has been minimized.

Show comment
Hide comment
@StFS

StFS Apr 7, 2015

Thanks for your reply.

Basically, we use AWS for some things but we also have our own EC2/S3 compatable private cloud setup (we're using https://qstack.com/).

Maybe I'm misunderstanding something with this. I know that the endpoints seem to differ between whether you're "talking" S3 or EC2. Basically I think I want to be able to point one of my profiles to a different "set" of endpoints. The aws cli tool works fine for our AWS account, but when I want to use it for our private cloud setup I always have to specify both --profile (to get the credentials right) and --endpoint-url (so that aws contacts our private cloud endpoint instead of the AWS ones).

StFS commented Apr 7, 2015

Thanks for your reply.

Basically, we use AWS for some things but we also have our own EC2/S3 compatable private cloud setup (we're using https://qstack.com/).

Maybe I'm misunderstanding something with this. I know that the endpoints seem to differ between whether you're "talking" S3 or EC2. Basically I think I want to be able to point one of my profiles to a different "set" of endpoints. The aws cli tool works fine for our AWS account, but when I want to use it for our private cloud setup I always have to specify both --profile (to get the credentials right) and --endpoint-url (so that aws contacts our private cloud endpoint instead of the AWS ones).

@StFS

This comment has been minimized.

Show comment
Hide comment
@StFS

StFS Apr 8, 2015

@kyleknap Am I totally out there with this?

StFS commented Apr 8, 2015

@kyleknap Am I totally out there with this?

@kyleknap

This comment has been minimized.

Show comment
Hide comment
@kyleknap

kyleknap Apr 8, 2015

Member

@StFS
Your use case makes sense to me. I am not sure how many users have a similar use case. I think if we were to add it as a feature we would be more inclined to do it per service, instead of setting a global value for endpoint url. The good news is that if we were to add this feature you would be able to configure the endpoint url for each service per profile (so you can still just use --profile and not have to include --endpoint-url).

Marking as feature request.

Member

kyleknap commented Apr 8, 2015

@StFS
Your use case makes sense to me. I am not sure how many users have a similar use case. I think if we were to add it as a feature we would be more inclined to do it per service, instead of setting a global value for endpoint url. The good news is that if we were to add this feature you would be able to configure the endpoint url for each service per profile (so you can still just use --profile and not have to include --endpoint-url).

Marking as feature request.

@StFS

This comment has been minimized.

Show comment
Hide comment
@StFS

StFS Apr 10, 2015

Ok thanks for the response again.

I'm a bit confused about these endpoints. It seems that for AWS the endpoint may change depending on what region you're using and also what service you're using.

For us, we would only need to specify ec2_endpoint and s3_endpoint, is that what you mean by "per service"? Are there other services than EC2 and S3? I think I've seen something involving user management as well but I'm not quite sure.

StFS commented Apr 10, 2015

Ok thanks for the response again.

I'm a bit confused about these endpoints. It seems that for AWS the endpoint may change depending on what region you're using and also what service you're using.

For us, we would only need to specify ec2_endpoint and s3_endpoint, is that what you mean by "per service"? Are there other services than EC2 and S3? I think I've seen something involving user management as well but I'm not quite sure.

@jerrycattell

This comment has been minimized.

Show comment
Hide comment
@jerrycattell

jerrycattell May 29, 2015

We have a similar use case (S3 alternative in Canada), and we would love to be able to configure the endpoint-url in the config file. Even if that means having a separate profile per service (currently it is only used for object storage), it would allow us to use the same commands in all of our environments.

jerrycattell commented May 29, 2015

We have a similar use case (S3 alternative in Canada), and we would love to be able to configure the endpoint-url in the config file. Even if that means having a separate profile per service (currently it is only used for object storage), it would allow us to use the same commands in all of our environments.

@kennu

This comment has been minimized.

Show comment
Hide comment
@kennu

kennu Oct 25, 2015

I think this would also be useful for configuring a "local" profile for accessing DynamoDB Local. Currently you have to write something like

aws --profile local --endpoint-url http://localhost:8000 dynamodb list-tables

kennu commented Oct 25, 2015

I think this would also be useful for configuring a "local" profile for accessing DynamoDB Local. Currently you have to write something like

aws --profile local --endpoint-url http://localhost:8000 dynamodb list-tables
@viniciussbs

This comment has been minimized.

Show comment
Hide comment
@viniciussbs

viniciussbs Nov 26, 2015

Same case here, @kennu. I have to set endpoint-url everytime I need to run some command on dynamodb-local.

I would love to set http://localhost:8000 as my default endpoint-url for dynamodb and set a blank endpoint-url to an production profile. It's more error-proof.

viniciussbs commented Nov 26, 2015

Same case here, @kennu. I have to set endpoint-url everytime I need to run some command on dynamodb-local.

I would love to set http://localhost:8000 as my default endpoint-url for dynamodb and set a blank endpoint-url to an production profile. It's more error-proof.

@mathurasri

This comment has been minimized.

Show comment
Hide comment
@mathurasri

mathurasri May 11, 2016

How to set dyanmodb local endpoint-url ? I am using ubuntu OS.

mathurasri commented May 11, 2016

How to set dyanmodb local endpoint-url ? I am using ubuntu OS.

@nicolas2bert nicolas2bert referenced this issue Jun 23, 2016

Closed

aws-cli #78

@shravanpn7

This comment has been minimized.

Show comment
Hide comment
@shravanpn7

shravanpn7 Sep 16, 2016

+1 for feature-request.

I am currently using aws cli to access minio.io server and specifying --endpoint-url every time is a pain.

shravanpn7 commented Sep 16, 2016

+1 for feature-request.

I am currently using aws cli to access minio.io server and specifying --endpoint-url every time is a pain.

@slejworks

This comment has been minimized.

Show comment
Hide comment
@slejworks

slejworks Sep 22, 2016

I need this also, connecting to local dnyamodb and I don't want to specify the endpoint for each cmd, simple entry into the config will work for me so I can concentrate on the command. thanks.

slejworks commented Sep 22, 2016

I need this also, connecting to local dnyamodb and I don't want to specify the endpoint for each cmd, simple entry into the config will work for me so I can concentrate on the command. thanks.

@christians3

This comment has been minimized.

Show comment
Hide comment
@christians3

christians3 Oct 4, 2016

+1 for this feature of allowing endpoint-url in the configuration file.

christians3 commented Oct 4, 2016

+1 for this feature of allowing endpoint-url in the configuration file.

@philloooo

This comment has been minimized.

Show comment
Hide comment
@philloooo

philloooo Oct 4, 2016

if accepting endpoint-url on the top level doesn't make sense, what about accepting it on service level?
eg:

[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  endpoint-url = test.org

philloooo commented Oct 4, 2016

if accepting endpoint-url on the top level doesn't make sense, what about accepting it on service level?
eg:

[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  endpoint-url = test.org
@jhalterman

This comment has been minimized.

Show comment
Hide comment
@jhalterman

jhalterman Oct 20, 2016

Similar use case, pointing to a Eucalyptus cloud. Would be nice to have a per-service config for endpoint-url.

jhalterman commented Oct 20, 2016

Similar use case, pointing to a Eucalyptus cloud. Would be nice to have a per-service config for endpoint-url.

@lincolnthomas

This comment has been minimized.

Show comment
Hide comment
@lincolnthomas

lincolnthomas Oct 20, 2016

How about also for the convenience of not having to type the --endpoint on the cmdline?

lincolnthomas commented Oct 20, 2016

How about also for the convenience of not having to type the --endpoint on the cmdline?

@wrsuarez

This comment has been minimized.

Show comment
Hide comment
@wrsuarez

wrsuarez commented Nov 16, 2016

+1

@jmarshall

This comment has been minimized.

Show comment
Hide comment
@jmarshall

jmarshall Nov 17, 2016

We have a third-party library that (as a minor aspect of its remote data access facilities) itself parses ~/.aws/credentials in order to construct the HTTP headers to access data in S3 buckets. Similarly to this issue, our users want to access their own S3-compatible data stores (cf samtools/htslib#436) and it would be useful if there were a standard well-known configuration file setting name for this purpose.

jmarshall commented Nov 17, 2016

We have a third-party library that (as a minor aspect of its remote data access facilities) itself parses ~/.aws/credentials in order to construct the HTTP headers to access data in S3 buckets. Similarly to this issue, our users want to access their own S3-compatible data stores (cf samtools/htslib#436) and it would be useful if there were a standard well-known configuration file setting name for this purpose.

@jedi4ever

This comment has been minimized.

Show comment
Hide comment
@jedi4ever

jedi4ever commented Nov 21, 2016

👍

@tomviner

This comment has been minimized.

Show comment
Hide comment
@tomviner

tomviner Nov 23, 2016

We also have this use case, except our S3-like-service has a different authorisation scheme (a simple authorization: <token> header). How are people using aws-cli to authorise against their S3-like-services?

tomviner commented Nov 23, 2016

We also have this use case, except our S3-like-service has a different authorisation scheme (a simple authorization: <token> header). How are people using aws-cli to authorise against their S3-like-services?

@wbingli

This comment has been minimized.

Show comment
Hide comment
@wbingli

wbingli Jan 2, 2017

Contributor

Check out this awscli plugin to set endpoint on profile: https://github.com/wbinglee/awscli-plugin-endpoint

Once you install(pip install awscli-plugin-endpoint) and config plugin(aws configure set plugins.endpoint awscli_plugin_endpoint), you can set endpoint in your profile per service as following:

[profile local]
dynamodb =
    endpoint_url = http://localhost:8000

Now you can run command with this endpoint for this service with only profile name:

aws dynamodb list-tables --profile local

See more details on project homepage(https://github.com/wbinglee/awscli-plugin-endpoint).

Let me know your feedback, :)

Contributor

wbingli commented Jan 2, 2017

Check out this awscli plugin to set endpoint on profile: https://github.com/wbinglee/awscli-plugin-endpoint

Once you install(pip install awscli-plugin-endpoint) and config plugin(aws configure set plugins.endpoint awscli_plugin_endpoint), you can set endpoint in your profile per service as following:

[profile local]
dynamodb =
    endpoint_url = http://localhost:8000

Now you can run command with this endpoint for this service with only profile name:

aws dynamodb list-tables --profile local

See more details on project homepage(https://github.com/wbinglee/awscli-plugin-endpoint).

Let me know your feedback, :)

jmarshall added a commit to samtools/htslib that referenced this issue Jan 9, 2017

Support custom S3 endpoint host_base setting (in .s3cfg)
Allow the user to specify an endpoint other than s3.amazonaws.com.
This can be set using ~/.s3cfg's host_base setting (only; we ignore
host_bucket); when there's a blessed setting key for .aws/credentials,
we'll support it there too (perhaps endpoint_url; cf aws/aws-cli#1270).

Fixes (part of) #436.
@RichardBronosky

This comment has been minimized.

Show comment
Hide comment
@RichardBronosky

RichardBronosky Feb 13, 2017

Contributor

Just to be clear, this is not only a benefit for people who are running AWS-compatible competitive products. It would also benefit those of us who are trying to use Amazon tools for local/offline development. http://stackoverflow.com/a/32260680/117471 is an example with DynamoDB. Using different profiles allows us to run our code with different configs for different environments. Not allowing endpoint_url to be specified in ~/.aws/config means that we have to build logic into our apps to follow a different (although small) path depending on the environment. That is something that should be avoided.

Contributor

RichardBronosky commented Feb 13, 2017

Just to be clear, this is not only a benefit for people who are running AWS-compatible competitive products. It would also benefit those of us who are trying to use Amazon tools for local/offline development. http://stackoverflow.com/a/32260680/117471 is an example with DynamoDB. Using different profiles allows us to run our code with different configs for different environments. Not allowing endpoint_url to be specified in ~/.aws/config means that we have to build logic into our apps to follow a different (although small) path depending on the environment. That is something that should be avoided.

@fatihtekin

This comment has been minimized.

Show comment
Hide comment
@fatihtekin

fatihtekin Dec 3, 2017

you can do below work around

from boto3 import Session
Session.client.defaults = (None, None, False, None, 'http://localhost:4575', None, None, None, None)

fatihtekin commented Dec 3, 2017

you can do below work around

from boto3 import Session
Session.client.defaults = (None, None, False, None, 'http://localhost:4575', None, None, None, None)
@aldanor

This comment has been minimized.

Show comment
Hide comment
@aldanor

aldanor Dec 3, 2017

@fatihtekin The whole question here is how to not do this but rather have it picked up from config.

aldanor commented Dec 3, 2017

@fatihtekin The whole question here is how to not do this but rather have it picked up from config.

@fatihtekin

This comment has been minimized.

Show comment
Hide comment
@fatihtekin

fatihtekin Dec 3, 2017

@aldanor i totally agree that is why it is a workaround if someone needs for testing purposes
especially without changing the lambda implementation

fatihtekin commented Dec 3, 2017

@aldanor i totally agree that is why it is a workaround if someone needs for testing purposes
especially without changing the lambda implementation

@jhart-r7

This comment has been minimized.

Show comment
Hide comment
@jhart-r7

jhart-r7 Jan 24, 2018

+1 for this feature. Given the rise of non-AWS providers providing AWS-like functionality with compatible APIs, having this functionality would be very useful.

jhart-r7 commented Jan 24, 2018

+1 for this feature. Given the rise of non-AWS providers providing AWS-like functionality with compatible APIs, having this functionality would be very useful.

@tfili

This comment has been minimized.

Show comment
Hide comment
@tfili

tfili commented Jan 30, 2018

+1

@ASayre

This comment has been minimized.

Show comment
Hide comment
@ASayre

ASayre Feb 6, 2018

Contributor

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at : https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168307-ability-to-specify-endpoint-url-in-profile

Contributor

ASayre commented Feb 6, 2018

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at : https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168307-ability-to-specify-endpoint-url-in-profile

@ASayre ASayre closed this Feb 6, 2018

@ankon

This comment has been minimized.

Show comment
Hide comment
@lpriima

This comment has been minimized.

Show comment
Hide comment
@lpriima

lpriima commented Apr 6, 2018

+1

@jamesls

This comment has been minimized.

Show comment
Hide comment
@jamesls

jamesls Apr 6, 2018

Member

Based on community feedback, we have decided to return feature requests to GitHub issues.

Member

jamesls commented Apr 6, 2018

Based on community feedback, we have decided to return feature requests to GitHub issues.

@abodnar

This comment has been minimized.

Show comment
Hide comment
@abodnar

abodnar Apr 19, 2018

This would be incredibly useful for those that are interacting with GovCloud.

abodnar commented Apr 19, 2018

This would be incredibly useful for those that are interacting with GovCloud.

@byterussian

This comment has been minimized.

Show comment
Hide comment
@byterussian

byterussian commented Apr 23, 2018

+1

1 similar comment
@sparrowjack63

This comment has been minimized.

Show comment
Hide comment
@sparrowjack63

sparrowjack63 commented May 17, 2018

+1

@mojodna

This comment has been minimized.

Show comment
Hide comment
@mojodna

mojodna Jun 13, 2018

This would also be helpful when working with Snowball Edges.

mojodna commented Jun 13, 2018

This would also be helpful when working with Snowball Edges.

@chaijunkun

This comment has been minimized.

Show comment
Hide comment
@chaijunkun

chaijunkun Jul 12, 2018

Yes, It is awesome that so many people expect this feature. I am using third party s3 provider, and the quality of command line tools which is developed by third party is not so good. we are looking forward amazon official version that satisfies our requirement.

well, the other way to implement this feature indirectly is:
set an alias for your general command. such as:
modify your ~/.bash_profile
add following line:
alias awsoss='aws s3 --endpoint-url=http://127.0.0.1:9000'
then you can use awsoss to replace aws command for accessing s3.

if you have multiple endpoints, you can also use:
alias awsoss='aws s3 --endpoint-url=$OSS_ENDPOINT'
alias awsossapi='aws s3api --endpoint-url=$OSS_ENDPOINT'

use: export OSS_ENDPOINT=xxx to switch different endpoints
then, you can use awsoss and awsossapi to instead of "aws s3" and "aws s3api"

chaijunkun commented Jul 12, 2018

Yes, It is awesome that so many people expect this feature. I am using third party s3 provider, and the quality of command line tools which is developed by third party is not so good. we are looking forward amazon official version that satisfies our requirement.

well, the other way to implement this feature indirectly is:
set an alias for your general command. such as:
modify your ~/.bash_profile
add following line:
alias awsoss='aws s3 --endpoint-url=http://127.0.0.1:9000'
then you can use awsoss to replace aws command for accessing s3.

if you have multiple endpoints, you can also use:
alias awsoss='aws s3 --endpoint-url=$OSS_ENDPOINT'
alias awsossapi='aws s3api --endpoint-url=$OSS_ENDPOINT'

use: export OSS_ENDPOINT=xxx to switch different endpoints
then, you can use awsoss and awsossapi to instead of "aws s3" and "aws s3api"

@dan-bhakta

This comment has been minimized.

Show comment
Hide comment
@dan-bhakta

dan-bhakta Jul 13, 2018

+1 for this feature

dan-bhakta commented Jul 13, 2018

+1 for this feature

@cws-credsimple

This comment has been minimized.

Show comment
Hide comment
@cws-credsimple

cws-credsimple Jul 16, 2018

+1 from me as well. We'd really like to leverage per-service endpoint URLs for development and testing.

cws-credsimple commented Jul 16, 2018

+1 from me as well. We'd really like to leverage per-service endpoint URLs for development and testing.

@onraz

This comment has been minimized.

Show comment
Hide comment
@onraz

onraz Jul 23, 2018

+1 This is needed for local development/testing and Dockerized environments.

onraz commented Jul 23, 2018

+1 This is needed for local development/testing and Dockerized environments.

@AlbertoPeon

This comment has been minimized.

Show comment
Hide comment
@AlbertoPeon

AlbertoPeon commented Aug 14, 2018

+1 for this

@brandonsimpkins

This comment has been minimized.

Show comment
Hide comment
@brandonsimpkins

brandonsimpkins Aug 15, 2018

+1 for this feature.

This is will be a useful feature for interacting with EC2 instances on a Snowball Edge (which act like a local region).

To list the EC2 instances on a snowball I have to specify the endpoint even though I already have a AWS CLI Named Profile that has the rest of the Snowball Edge specific info (secret / access keys):

aws ec2 describe-instances --profile snowballEdge --endpoint http://${snowball_ip}:8008

brandonsimpkins commented Aug 15, 2018

+1 for this feature.

This is will be a useful feature for interacting with EC2 instances on a Snowball Edge (which act like a local region).

To list the EC2 instances on a snowball I have to specify the endpoint even though I already have a AWS CLI Named Profile that has the rest of the Snowball Edge specific info (secret / access keys):

aws ec2 describe-instances --profile snowballEdge --endpoint http://${snowball_ip}:8008

@eedwards-sk

This comment has been minimized.

Show comment
Hide comment
@eedwards-sk

eedwards-sk Aug 29, 2018

+1 to this and environment variable for the same (e.g. AWS_ENDPOINT or AWS_S3_ENDPOINT)

eedwards-sk commented Aug 29, 2018

+1 to this and environment variable for the same (e.g. AWS_ENDPOINT or AWS_S3_ENDPOINT)

@mrvickiee

This comment has been minimized.

Show comment
Hide comment
@mrvickiee

mrvickiee Oct 10, 2018

+1 for this feature

mrvickiee commented Oct 10, 2018

+1 for this feature

@dora-korpar

This comment has been minimized.

Show comment
Hide comment
@dora-korpar

dora-korpar commented Oct 22, 2018

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment