HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure #2473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
added
bug
fs/s3
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
OK. I'd thought these files were deleted. |
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
Outdated
Show resolved
Hide resolved
The addition of deprecated S3A configuration options in HADOOP-17318 triggered a reload of default (xml resource) configurations, which breaks tests which fail if there's a per-bucket setting inconsistent with test setup. Creating an S3AFS instance before creating the Configuration() instance for test runs gets that reload out the way before test setup takes place. Along with the fix, extra changes in the failing test suite to fail fast when marker policy isn't as expected, and to log FS state better. Change-Id: I9f58cf0d42c5ac98962b4a9fed9f3ff93ef182ac
Change-Id: Ic48f762aea7552ec1d204298326ad3f2c2b749ea
+Rebase to trunk to get rid of the license warnings Change-Id: I679ea2711ce832700ef4aae0ce16008483ff40bc
steveloughran
force-pushed
the
s3/HADOOP-17385-ITestS3ADeleteCost-failure
branch
from
ebd8242 to
57f30e3
Compare
|
rebased to trunk to fix license warnings; fixed up error in JIRA ID. tested: s3 london |
This comment has been minimized.
This comment has been minimized.
bgaborg
requested changes
Nov 25, 2020
| FileSystem.getLocal(new Configuration()); | ||
| // instantiate an S3A FS here here to force deprecated key load through the | ||
| // static initializers. See: HADOOP-17385 | ||
| new S3AFileSystem(); |
There was a problem hiding this comment.
I think it would be cleaner if we could run the static initializers instead of creating an object without storing the reference. I know that there's a comment there stating that this is required because of a jira, but I don't see the point why would we create an object just to trigger side effects.
There was a problem hiding this comment.
I was about to say I don't know a way to trigger instantiation of static class initializers except by a new() call, but actually we can invoke any static method. How about I add a new static initializeClass() call and load it.
FWIW, what I'd really like is to be able to disable forced reload of configs, but I can't see a way to do that which wouldn't run the risk of damage
Rather than create and discard an instance, add a new static method to S3AFS and invoke it in test setup. This forces the load Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
This comment has been minimized.
This comment has been minimized.
|
thanks; will cp to branch-3.3 along with the PR which caused the regression. |
asfgit
pushed a commit
that referenced
this pull request
Nov 26, 2020
…2473). Contributed by Steve Loughran The addition of deprecated S3A configuration options in HADOOP-17318 triggered a reload of default (xml resource) configurations, which breaks tests which fail if there's a per-bucket setting inconsistent with test setup. Creating an S3AFS instance before creating the Configuration() instance for test runs gets that reload out the way before test setup takes place. Along with the fix, extra changes in the failing test suite to fail fast when marker policy isn't as expected, and to log FS state better. Rather than create and discard an instance, add a new static method to S3AFS and invoke it in test setup. This forces the load Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
…on failure (apache#2473). Contributed by Steve Loughran The addition of deprecated S3A configuration options in HADOOP-17318 triggered a reload of default (xml resource) configurations, which breaks tests which fail if there's a per-bucket setting inconsistent with test setup. Creating an S3AFS instance before creating the Configuration() instance for test runs gets that reload out the way before test setup takes place. Along with the fix, extra changes in the failing test suite to fail fast when marker policy isn't as expected, and to log FS state better. Rather than create and discard an instance, add a new static method to S3AFS and invoke it in test setup. This forces the load Note: this was committed in 7.1.8 with the tag CDPD-19868. Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The addition of deprecated S3A configuration options in HADOOP-17318
triggered a reload of default (xml resource) configurations, which breaks
tests which fail if there's a per-bucket setting inconsistent with test
setup.
Creating an S3AFS instance before creating the Configuration() instance
for test runs gets that reload out the way before test setup takes
place.
Along with the fix, extra changes in the failing test suite to fail
fast when marker policy isn't as expected, and to log FS state better.
Testing: S3 London with marker policy forced to override (which is why this specific test failed)