Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Cannot resolve role ARN when regional STS endpoint is configured #7418
Describe the bug
I think this must be related to this code: https://github.com/hashicorp/vault/blob/v1.2.2/builtin/credential/aws/backend.go#L252
As the error reports a different invalid region if you repeat the command, until eventually by luck it matches and the role can be created.
Note if you disable
With custom STS endpoint
Hi @hamishforbes , thanks for opening this issue.
I'm curious if setting
Could you take a look and maybe give it a try and see if it solves things for you?
Hi, Is that doco just talking about picking up the correct creds and region for the client to authenticate itself to AWS? Which is working, my Vault servers correctly pick up creds via the EC2 instance profile.
Setting the AWS_REGION variable hasn't made a difference to the ARN resolving problem.
I'm not sure if using
Should it not just try and parse the region out of the
Currently the request is signed using a random region from the current AWS partition (normal, gov, china)
There's also a related issue with Vault CLI and Vault server default behaviours.
The Vault CLI, with no additional configuration, looks up your local region from environment/metadata and signs using that.
This is less of a bug and more of a usability / doco thing I think?
IMO the client should default to using us-east-1 when logging in unless explicitly set to use another region (by passing
I think this makes more sense as a user, if you don't configure anything then it doesn't matter which region the server and client are in, they both sign using us-east-1 and everything works via the global STS endpoint.
Or the server should also lookup it's own region via metadata and use the local region endpoint + signing.
Currently if you don't configure anything any client outside of us-east-1 has to specify us-east-1 as the region, which is super confusing given I have zero resources in us-east-1!
Hey - I'm hitting this (random region selection) too as a result of adding
Same here - it makes no sense to me, but looks like the only way to work around the change in behaviour as things stand.
This particular issue only comes into play when Vault is trying to determine the AWS account ID corresponding to the credentials it is using. It needs to do this to ensure that the ARNs being bound are actually resolvable. (For example, if you pass a
I think the best way is what @tyrannosaurus-becks suggests -- namely, ask Vault operators to specify the AWS region and then use
I can see a couple other options, but I don't think they're as good:
A few other notes:
Actually, the global
In theory, yes, you're right. But, more practically speaking, why would you or anyone else ever do that? Is there a valid use case for this? By using a closer region, you can take advantage of lower latency and VPC endpoints. And, even if you wanted to do that, in my proposal, you could just override the region parameter in the client config to be
Can we move this discussion to a separate GitHub issue? I'm not trying to shut down debate here; I genuinely think you're raising real usability concerns and I'd love to continue the discussion about how to improve the usability and user experience. But the discussions around AWS STS endpoints is confusing enough and I think it'll be easier to keep these two discussions separate.
So having an
If you configure a signing region and an appropriate endpoint then clients in the same region will 'just work'.
But servers outside of
No, it was just a contrived scenario that would break, I think I was assuming it would just use
For sure, although I think if theres a configurable signing region then the disconnect between the 2 is less pronounced so maybe not needed
No. The client already has this parameter. The proposal would be to add this to the server.
No. Servers configured with a custom
Maybe? The client signing region would still need to match the server signing region, and the disconnect between those two isn't great. (One could imagine a helper to read the server-side STS signing region first before doing the client-side signing, but that configuration would need to be anonymously readable, because you need to sign a request to authenticate to Vault in the first place, and I'm not sure that's a great idea.) Which is why I think a separate issue to discuss how to improve the user experience would be great to make this better for everyone.
I think we're in decent accord here, and would support a PR to that effect. @joelthompson you're welcome to do it if you'd like. If not, I can take it on at my next opportunity, but I need to deliver some other unrelated items first so it will be at least a month or two before I can start.