HADOOP-16626. S3A ITestRestrictedReadAccess fails #1587
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
force-pushed
the
s3/HADOOP-16626-ITestRestrictedReadAccess
branch
from
aeb7768
to
3e4ae00
Compare
|
Testing s3a ireland |
steveloughran
force-pushed
the
s3/HADOOP-16626-ITestRestrictedReadAccess
branch
from
3e4ae00
to
9b479aa
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Looks good to me mostly, without fully understanding the problem which caused this (the resource loading bit unsetting configs, but what was being unset).
|
Which of course is entirely unexpected. Having a look at why things are failing for you, -Ds3guard -Dlocal had been fine for me. I had problems with this when changing the test because you need to share the same local store instance across the real fs and the restricted one -otherwise the restricted one's cache is empty, so it falls back to s3 checks. I may just change the test so that it requires s3guard + ddb set on the maven command line to run those guarded tests, and not worry about the local one at all. It's just complicating things too much and especially given the local store cannot be used in production |
|
oh, and it does work for me: -Dit.test=ITestRestrictedReadAccess -Ds3guard I am going to go to only running the guarded tests if -Ds3guard -Ddynamo is set |
|
latest patch skips all the s3guard test runs if the store is local; you must be running with DDB enabled for it to work. That is runs without anything or with only -Ds3guard will only run one of the tests; the other two should be reported as skipped. To get all three -run with -dynamo Made sure there were no DDB bindings in my auth keys files to verify that things are good |
-Force load the filesystems early on -And only add the contract resource if needed. -Only run the guarded tests if S3Guard is on according to the build. I had a predecessor which always used the Local store, but it was hard to set up -you need to share across FS instances-, and you could never guarantee that it worked the same way with DDB. That patching is still there -it's just not needed/used for the DDB test runs Change-Id: I79644ac264f74005775ff194d48f08fe951df0f1
This avoids trying to force all the DDB settings to be consistent and deal with the problem that the local store isn't coherent across FS instances. Change-Id: I50113be53d3d34c1748e498af2075137c3e3afd3
steveloughran
force-pushed
the
s3/HADOOP-16626-ITestRestrictedReadAccess
branch
from
208c98f
to
433f0b3
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
LGTM. +1 |
|
thx. will merge; then rebase #1601 on top and use the same explicit disabling of the metastore to ensure test runs always validate the unguarded path there too. |
|
merged. And I have learned some facts about Configuration that I didn't want to. Someone should update the javadocs there...or we add the notion of tombstone markers in the config :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix up test setup for the restricted access.
-Force load the filesystems early on
-And only add the contract resource if needed.
-Only run the guarded tests if S3Guard is on according to the build.
I had a predecessor which always used the Local store, but it was
hard to set up -you need to share across FS instances-, and you could
never guarantee that it worked the same way with DDB. That patching is
still there -it's just not needed/used for the DDB test runs
Change-Id: I79644ac264f74005775ff194d48f08fe951df0f1