update S3 pre-signed credentials logic

[bentsku]

Motivation

Currently in LocalStack, in order to use pre-signed URLs while validating the signature (with S3_SKIP_SIGNATURE_VALIDATION set to false), the user needs to set hardcoded credentials: test/test.

This is due to the fact that we did not retrieve the Secret Access Key linked to the sent Access Key Id, which is needed to take the incoming request server side, sign it as well and compare with the received signature.

We also used the TEST_AWS* credentials to generate the signature server side, which was a way for our test to always work as those are hardcoded.

Changes

• Implement a way to retrieve the Secret Access Key of the user by directly accessing the moto backend, as there are no public API to retrieve the secret access key for obvious security reasons
• however, as we don't create IAM users by default in LocalStack, we still need to rely on the hardcoded value of the secret access key to be test, so that we can properly validate pre-signed URL even the user does not provide credentials that actually exists in LocalStack.
• update the tests to use the hardcoded S3 constants instead of the test credentials

Note: as we can use random Access Keys in LocalStack without having them being created in IAM, most of the time we won't be able to retrieve the secret key, and will need to fallback to the test secret access key.

However, we now support passing proper existing credentials and having the handler properly validate them.

\cc @dfangl as I'm directly accessing IAM internals, not sure what is your opinion on this, and if it can be done any other way?

[github-actions]

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   3m 11s ⏱️
387 tests 336 ✅  51 💤 0 ❌
774 runs  672 ✅ 102 💤 0 ❌

Results for commit 792008e.

♻️ This comment has been updated with latest results.

[coveralls]

coverage: 83.872% (+0.01%) from 83.86%
when pulling 792008e on s3-presigned-upgrade
into db3c42f on master.

[github-actions]

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 22m 9s ⏱️ +52s
2 643 tests +1  2 396 ✅ +1  247 💤 ±0  0 ❌ ±0 
2 645 runs  +1  2 396 ✅ +1  249 💤 ±0  0 ❌ ±0 

Results for commit 792008e. ± Comparison against base commit db3c42f.

♻️ This comment has been updated with latest results.

[viren-nadkarni]

Q: Is it possible to use STS:GetSessionToken at this point and obtain the access key ID + secret access key programatically?

[bentsku]

I don't think so, I think GetSessionToken returns a set of temporary credentials which will be then be different than what was used to signed the request on the client side.

I think you would also need to create the client for STS with the Access Key ID and the Secret Access Key of the client used to sign the request?

[bentsku]

This made me think of @dfangl hackathon, where he must had to implement a similar logic to validate the signature of every request:
f758364

master...hackathon-sig

And I guess it was the same issue, about having credentials that are set in LocalStack.

So I'm just not sure about the reverse of the access key id <-> account id to access the store, or if I need to iterate in the same way.

[dfangl]

There seems to be no API for getting the secret key, so I think this is the best way for now :)

[viren-nadkarni]

Thanks for adding this @bentsku, another step towards improving our test strategy 🙌

[dfangl]

Only really looked at the IAM/STS stuff, but looks good! Sadly there is no better way as far as I know :)