Skip to content

HADOOP-16658 - S3A connector does not support including the token ren…#1664

Closed
pzampino wants to merge 1 commit intoapache:trunkfrom
pzampino:HADOOP-16658
Closed

HADOOP-16658 - S3A connector does not support including the token ren…#1664
pzampino wants to merge 1 commit intoapache:trunkfrom
pzampino:HADOOP-16658

Conversation

@pzampino
Copy link

…ewer in the token identifier

Ran 'mvn verify' with endpoint s3.us-east-1.amazonaws.com
Two tests failed, but passed when run individually:

  • ITestS3AConfiguration#testAutomaticProxyPortSelection
  • ITestS3AInconsistency#testGetFileStatus

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for renewer to not be null but actually be empty?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible for it to be empty, in which case the renewer Text object will be "empty", and that case is checked by yarn.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • need to make sure that null results in renewer == new Text()
  • add some spaces round the !=

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As these are public methods, are you sure that we don't need to preserve the old signature as well for possible extensions? Does this appear in a public release yet at all?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not yet appeared in a public release; Steve actually suggested that I remove the previously-existing signatures, which I had marked as deprecated initially.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no ASF release. The only people using this outside the hadoop-aws jar are you; I have plans to do other incompatible changes soon. Sorry.

@steveloughran steveloughran self-assigned this Oct 22, 2019
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a message for better diags, e.g. "renewer"

Copy link
Contributor

@steveloughran steveloughran Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add message "renewer in " + ids

Copy link
Contributor

@steveloughran steveloughran Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add message "renewer in " + ids

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message "renewer in " + ids

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the renewer may now be null. should we add a new Text() if a null comes in?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an assert in this test case to verify load of renewer

@steveloughran
Copy link
Contributor

Code LGTM; some minor comments about tests

I'd like AbstractS3ATokenIdentifier to create a Text() if a null renewer was passed in; this is consistent with the existing code.

@pzampino
Copy link
Author

I've made the suggested changes, and re-ran the tests (which passed again).

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 1777 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 1119 trunk passed
+1 compile 38 trunk passed
+1 checkstyle 28 trunk passed
+1 mvnsite 40 trunk passed
+1 shadedclient 820 branch has no errors when building and testing our client artifacts.
+1 javadoc 30 trunk passed
0 spotbugs 60 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 59 trunk passed
_ Patch Compile Tests _
+1 mvninstall 33 the patch passed
+1 compile 29 the patch passed
+1 javac 29 the patch passed
-0 checkstyle 20 hadoop-tools/hadoop-aws: The patch generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9)
+1 mvnsite 33 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 792 patch has no errors when building and testing our client artifacts.
+1 javadoc 26 the patch passed
+1 findbugs 70 the patch passed
_ Other Tests _
+1 unit 86 hadoop-aws in the patch passed.
+1 asflicense 33 The patch does not generate ASF License warnings.
5119
Subsystem Report/Notes
Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/artifact/out/Dockerfile
GITHUB PR #1664
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f23e524c59ac 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / a901405
Default Java 1.8.0_222
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/testReport/
Max. process+thread count 472 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

LGTM; thanks for changes

+1 and merged to trunk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants