Parse region and service from AWS Endpoint

[matthewmueller]

Hi there,

I'm looking for a way to parse the region and service from an AWS string. I've looked at this: http://docs.aws.amazon.com/general/latest/gr/rande.html and while it seems pretty standardized at this point, there are some exceptions.

I'm wondering if there's some way to do this in the library:

service, region, err := endpoints.Parse("s3.eu-central-1.amazonaws.com")
// service: s3, region: eu-central-1

service, region, err := endpoints.Parse("s3-us-west-2.amazonaws.com")
// service: s3, region: us-west-2

With the goal being to be able to use that information to sign a request.

[xibz]

Hello @matthewmueller, thank you for reaching out to us. Questions like these are best suited for gitter and stackoverflow as you'll get feedback from the overall community. You can iterate through regions with the example here. You may be able to use that to your advantage without needing to parse the endpoint directly. Let me know if you any issues with that!

[jasdel]

In addition @matthewmueller once the endpoint is constructed its generally treated as a black box by the SDK. Primarily, because there is no strict format that is required. Some services multiple different endpoint formats such as S3 like you referenced in the example. But other services have no region component and are considered global such as sts.amazonaws.com.

What is the use case that you're looking to solve? I'm curious if there is another way to address the problem without parsing the endpoints URLs.

[matthewmueller]

Thanks for the feedback guys. I guess this is more of a feature request then. I'll definitely try gitter/stackoverflow next time something comes up.

@jasdel I want to create a signing proxy so I can do something like this:

curl https://AK123:S123@proxy.com/s3-us-west.amazonaws.com/my-bucket/123.jpg

or more generally:

curl https://${accessId}:${secretKey}@proxy.com/${service}/${path}

As far as I can tell, being able to parse the path is required to sign the request:

signer.Sign(...)

[jasdel]

Thanks for the update @matthewmueller. I strongly recommend not passing the AWS access key and secret in the URL. These credentials could easily be logged by the proxy or intermediate host unexpectedly leaking the credentials.

What is the motivation behind wanting to do a remote signer with credentials passed to it? Would it be viable for the proxy to maintain the credentials internally? This would reduce the risk of credentials being leaked in transit.

[jasdel]

Hi @matthewmueller since parsing the endpoint URL is not supported by the SDK, and there isn't a definitive format for the URL hostname I don't think it would be a good idea for the SDK to bake in this functionality which may break based on service hostname patterns.

[matthewmueller]

I strongly recommend not passing the AWS access key and secret in the URL. These credentials could easily be logged by the proxy or intermediate host unexpectedly leaking the credentials.

That's a really good point. This was something internal, but you're right, it's flawed from the outset.

Since parsing the endpoint URL is not supported by the SDK, and there isn't a definitive format for the URL hostname I don't think it would be a good idea for the SDK to bake in this functionality which may break based on service hostname patterns.

Actually, this is the reason I think the SDK should handle it instead of 3rd party libraries. You guys can stay up-to-date with any hostname changes that may come our way.

[jasdel]

@matthewmueller thanks for the feedback. At 106 services most will follow the general pattern, but the SDKs do not have a contract on how these endpoints in their various forms and level's of opaqueness could be deconstructed. The endpoints are initially treated opaquely once they are constructed allowing service teams the flexibility to create endpoints based on the service's use case and needs.

Services can create endpoints that do not match the general format. In some cases such as IoT DataPlane the endpoint can be completely opaque. IoT DataPlane uses an endpoint provided by IoT's DescribeEndpoint API call. The format of this endpoint is opaque. Other services such as S3 have several components that optionally can make up the endpoint such as <bucket>, accelerate, dualstack, and optionally no region for case us-external-1.

In a proxy/remote signer usecase would it be feasible to extract the region and service name from the SDK's request before it is sent to the proxy? Including the region as a query parameter.

[matthewmueller]

Ahh okay, now I have a better idea of the scope of this issue now and why something like this wouldn't be in the SDK. Thanks for explaining that in more detail.

I was sort of hoping to just have this signing function sitting on API gateway backed by a lambda function that signs requests for all AWS services. My use-case for this was a serverless RPM repository and so in this case I needed to parse the request on the proxy server itself and then fill in the appropriate fields (id, secret, region, service) to the SDK and return a signature along with the URL. I can't remember exactly now, but I think the discrepancies in the api url formats made it not really possible to do this in a generic way.

For inspiration, this was the package I was looking for: https://github.com/mhart/aws4. It looks like the author manually maintains the different URL formats. I was thinking there could be some help from the AWS SDK(s) in that respect.

For now I ended up scaling down my ambitions and just baked in S3 signing. I'd love to make this a bit more generic so I can call it from the browser or cURL easily. I think you're right that I should bake it into the lambda itself, but even internally, you'd need something like that node repo if you want to do this generically. Anyway, thanks for your thoughtful responses!

[jasdel]

Glad to help. You might also want to checkout the SDK's gitter channel where several users of the Go SDK will ask questions and discuss designs with each other. In addition there in the Go slack #AWS channel which commonly will discuss AWS use cases. If you're not already apart of the Go slack you can join via https://invite.slack.golangbridge.org/