HBASE-24280 TestSecureRESTServer started failing in nightlies for Had… by stoty · Pull Request #1603 · apache/hbase

stoty · 2020-04-29T08:15:50Z

…oop3

revert to using -Dhadoop.profile

…oop3 revert to using -Dhadoop.profile

Apache-HBase · 2020-04-29T08:26:15Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console in case of problems.

Apache-HBase · 2020-04-29T08:26:27Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console in case of problems.

Apache-HBase · 2020-04-29T08:26:27Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console in case of problems.

Apache-HBase · 2020-04-29T08:27:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 36s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
		_ Other Tests _
		2m 20s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#1603
Optional Tests
uname	Linux 4fd09b79a865 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f46dba8`
Max. process+thread count	52 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-29T08:28:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 35s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	shellcheck	0m 2s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
		_ Other Tests _
+0 🆗	asflicense	0m 0s	ASF License check generated no output?
		2m 34s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1603
Optional Tests	dupname asflicense shellcheck shelldocs
uname	Linux 81de352c2b3b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f46dba8`
Max. process+thread count	52 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-29T08:28:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 30s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
		_ Other Tests _
		2m 44s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1603
Optional Tests
uname	Linux dd3b522e7e93 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f46dba8`
Max. process+thread count	52 (vs. ulimit of 12500)
modules	C: U:
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1603/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

binlijin

+1

joshelser

Looks good to me. I'd have to go digging into why activating via system property and not profile is important in Maven (I remember there being a reason).

I want to wait for Duo to come back though, since this is essentially undo'ing a change of his from 2 weeks ago.

saintstack · 2020-04-29T16:16:20Z

Fixed by HBASE-24215 ?

stoty · 2020-04-29T16:22:26Z

HBASE-24215 may or may not fix the test error (I did not check),
but specifying profiles with -P definitely doesn't work with the HBase (and most non-trivial) maven profile setup.

See the JIRA for details on how it doesn't work.

saintstack · 2020-04-29T16:25:32Z

@stoty ok. Thanks. Maybe issue needs renaming then? Good work.

saintstack

Activation is basic:

 <profile>
   <id>hadoop-2.0</id>
   <activation>
     <property>
       <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
       <!--h2--><name>!hadoop.profile</name>
     </property>
   </activation>

Wasn't there a reason for using -P rather than this -Dhadoop.profile in the issue that made the change?

stoty · 2020-04-29T16:40:00Z

From what I gather, The problem was that in 3.0, only Hadoop 3 is supported, and it was part of the solution to make hadoop3 the default.

However, in a later patch it was solved by setting property to 3.0 by default, which is compatible with the branch-2 profile activation scheme.

saintstack · 2020-04-29T20:38:37Z

So, this patch is not fit given hbase-personality is used on all branches as Nick points out up on on the issue (master doesn't have the hadoop.version property).

Looking at master, it is trying to do profile AND property -- -P and -D -- but I'm not sure that activation stuff works for profiles.

 <profile>
   <id>hadoop-3.0</id>
   <activation>
     <property><name>!hadoop.profile</name></property>
   </activation>
   <properties>

...

Thinking on what this issue has turned up, let me go back to HBASE-24215 . I'm now of the opinion that the (painful) 'fix' there actually would not be needed if we didn't have both hadoop2 and hadoop3 profiles activated as you found here (no wonder I was having trouble repro'ing what was happening up on jenkins). Let me put this comment up in JIRA. Better suited there.

saintstack · 2020-05-12T23:26:32Z

Closing because the JIRA is resolved. Good stuff in here toward figuring out what root issue was around this test failing. Thanks @stoty

HBASE-24280 TestSecureRESTServer started failing in nightlies for Had…

8853413

…oop3 revert to using -Dhadoop.profile

binlijin approved these changes Apr 29, 2020

View reviewed changes

joshelser approved these changes Apr 29, 2020

View reviewed changes

saintstack reviewed Apr 29, 2020

View reviewed changes

saintstack closed this May 12, 2020

Conversation

stoty commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

Apache-HBase commented Apr 29, 2020

Uh oh!

binlijin left a comment

Choose a reason for hiding this comment

Uh oh!

joshelser left a comment

Choose a reason for hiding this comment

Uh oh!

saintstack commented Apr 29, 2020

Uh oh!

stoty commented Apr 29, 2020

Uh oh!

saintstack commented Apr 29, 2020

Uh oh!

saintstack left a comment

Choose a reason for hiding this comment

Uh oh!

stoty commented Apr 29, 2020

Uh oh!

saintstack commented Apr 29, 2020

Uh oh!

saintstack commented May 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants