Implement AWS Signature V4a for multi-region access point usage

[jwineinger]

AWS S3 Multi-region Access Points (MRAP) offer a single endpoint that can serve files from S3 buckets in different regions, optimized for latency and failing over in the event of a regional outage. As such, we're interested in using them for serving OPA bundles.

What is the underlying problem you're trying to solve?

For MRAPs to serve requests from multiple regions, the requests must be signed with AWS Signature Version 4a (SigV4a). Currently OPA only signs requests using AWS Signature V4.

AWS credentials retrieved from the global STS endpoint cannot be used for v4a; you must get credentials from a regional STS endpoint. I believe AWS SDKs are or have switched to default to regional endpoints, but this might need to be taken into account as well.

Describe the ideal solution

Ideally OPA would sign using V4 when hitting a service in a single region, such as a single S3 bucket, but then use V4a when requests could be made to a multi-region AWS service like MRAP. The docs on V4a are not great but it seems that the SDKs and algorithm can take a set of regions to sign for. This can be a list of 1+ regions, or *. So perhaps if the regions set is size 1, sign with V4 and sign with V4a if it has more than 1 or is *.

Describe a "Good Enough" solution

Alternatively, you could set the signing version to v4a in the OPA config, and it would set the region header to *.

Additional Context

It seems that OPA has its own implementation of v4 signing. Rather than reimplement v4a, I looked to see if the AWS go SDK could be used to sign requests. The aws-sdk-go-v2 does have a v4a signer, but it is under an internal package, and so not available. I opened an issue there to see if they could export it, but it doesn't appear that's going to happen anytime soon (aws/aws-sdk-go-v2#1935).

That repo does use the Apache 2 license, so it seems like that code could be copied into a public module somewhere and used by OPA, if that's deemed an acceptable course.

[anderseknert]

Not 100% sure I understood the reasoning provided here, but if the API is still subject to change, it would seem wise to wait until it's publicly exposed before implementing in OPA, or WDYT?

[jwineinger]

Same; I'm not sure what to think. To add to the confusion, they have this repo as well https://github.com/aws-samples/sigv4a-signing-examples. Seems that the signing algorithm is published and usable (required for MRAP), but I guess they haven't standardized an SDK interface? And yet they have implemented the algo in a few SDKs, and the AWS CLI does it as well. Perhaps the go SDK dev team doesn't want to expose their implementation publicly, while the teams for the other SDKs don't have the same hangups.

Personally, I'd like to get support for this into OPA, and since AWS won't commit to a date we'd just have to move forward. We could do a custom implementation of the new signing algo (as OPA does for the v4 signing stuff), or pull out their current code and use it. To that end, I do have a private repo that extracts the v4a signing stuff from their go SDK repo, and I was able to get OPA working with that on the "good enough" solution above. I can see about making a PR but I'll have a couple of employer hoops to jump through first.

[anderseknert]

Sounds like a way forward! I think we keep to our own implementation in order to avoid depending on their lib, which is known to update very frequently, and we'd only use like 0.05% of its functionality. This is especially important for people using OPA as a library, as they generally don't like to get a whole bunch of libraries included transitively.

CC @philipaconrad as you've worked with this recently.

[philipaconrad]

@anderseknert is correct: As far as I know, we roll our own implementation to avoid having more external dependencies. We also do not support all of the variations on Sig v4, just the most basic single-chunk, header-based auth method.

This works great for the new providers.aws.sign_req builtin (landed in PR #5376), as well as for the current REST plugin's S3 support.

@jwineinger I suspect a good way to handle this would be to duplicate the v4 signing stuff under internal/providers/aws.go, and then modify the algorithm to account for v4a's differences. The main user for v4a signing will probably be the REST plugin. With a new config file option or two, I could see it being straightforward to deploy too.