Force US_EAST_1 for S3 listBuckets call #66
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdedetrich
requested review from
jrudolph,
raboof,
gmethvin,
He-Pin,
nvollmar,
seglo and
pjfanning
|
Related tests (i.e. S3) passed |
|
@pjfanning What are your views for merging this now vs 1.1.x? |
|
if it doesn't work without this fix, merge it now |
|
Cool, I'll wait for a week or so to see if anyone else has any objections and then I will merge |
|
So I am merging this since no one objected. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So this is a bug which I discovered after I added this feature (akka/alpakka#2899) to Alpakka. I didn't manage to fix this in Alpakka because they changed their license to BSL before I discovered it.
Essentially, if you are using the list buckets call from a configured region that is not
US_EAST_1it will fail. As noted this only occurs in AWS and not Minio (Minio is a 3rd party implementation of AWS). I have added a test to confirm this behaviour, i.e. if you revertHttpRequests.listBuckets(s3Headers.headers), Some(Region.US_EAST_1))toHttpRequests.listBuckets(s3Headers.headers))the test will fail. Unfortunately in order to replicate this you need an actual AWS account to test against (I personally used a company test account to verify this).Regarding whether this should be put into 1.1.x or 1.0.x there are arguments both ways. In the current state the
listBucketscall will simply just fail to work if you are using a configured nonUS_EAST_1region. The only way to work around this is to manually override the attribute (i.e. https://github.com/aiven/guardian-for-apache-kafka/blob/9fce5561479d9800d82583e68b2ff1f0191d46c6/core-s3/src/test/scala/io/aiven/guardian/kafka/s3/Main.scala#L83-L92). If you do this manual workaround in conjunction with this bug fix there is no difference, where as if you don't have this bug fix it wont work at all. So in summary while it is a behavioural change, its very obvious and hence black and white what the change is and most critically there won't be any conflicts if users implemented a manual workaround (like the one I previously mentioned). That being said I have no issues if the community agrees to target this for 1.1.x