HBASE-24150 Allow module tests run in parallel #1464
Conversation
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow.
|
(!) A patch to the testing environment has been detected. |
2 similar comments
|
(!) A patch to the testing environment has been detected. |
|
(!) A patch to the testing environment has been detected. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
dev-support/hbase-personality.sh
Outdated
| @@ -140,7 +140,10 @@ function personality_modules | |||
|
|
|||
| clear_personality_queue | |||
|
|
|||
| extra="-DHBasePatchProcess" | |||
| # At a few points, hbase modules can run their test suites in parallel | |||
There was a problem hiding this comment.
Not just parallel test suites, but everything. With this change, maven will run up to 2 concurrent modules when the dependency graph permits. I imaging this will be a nice general benefit.
One thing to note though is that some plugins don't support multithreading. IIRC, surefire-reports was one such plugin, and would print warnings in the build. Looks like this patch didn't actually trigger any maven invocation, so I cannot point you at the message. A simple mvn clean package -DskipTests should be enough to get the message printed.
Since we use Jenkins to aggregate test results, I suggest we disable/remove surefire-reports (this was something I discussed with @busbey sometime back, but neither of us have gotten to it).
There was a problem hiding this comment.
Let me change the comment.
I tried running the mvn clean ... above to find the message you refer to but too blind to see it. Let me add change in pom so its generated here.
On disabling the surefire reporting, you want that in this commit or on a follow-on? Is it this you are referring too?
maven-surefire-report-plugin
There was a problem hiding this comment.
Yeah that's the plugin. Weirdly, now I try but I don't see the warning message. Hmm..
Running with -T1.0C I get a failure in
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (check-aggregate-license) @ hbase-shaded-client ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:
Couldn't evaluate condition: File license = new File("/Users/ndimiduk/repos/apache/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE");
// Beanshell does not support try-with-resources,
// so we must close this scanner manually
Scanner scanner = new Scanner(license);
while (scanner.hasNextLine()) {
if (scanner.nextLine().startsWith("ERROR:")) {
scanner.close();
return false;
}
}
scanner.close();
return true;
There was a problem hiding this comment.
Yeah, I see those WARNINGs locally too. Another issue?
There was a problem hiding this comment.
For me it's not just a warning, it breaks the build. Yes another issue, but if it consistently breaks runs using -T, it would need to be a blocker of this one.
| @@ -125,7 +125,7 @@ fi | |||
| # See http://hbase.apache.org/book.html#maven.release | |||
|
|
|||
| echo "Maven details, in case our JDK doesn't match expectations:" | |||
| mvn --version --offline | tee "${working_dir}/maven_version" | |||
| mvn --threads=2 --version --offline | tee "${working_dir}/maven_version" | |||
There was a problem hiding this comment.
There's no value in --threads for grabbing the version info, is there?
There was a problem hiding this comment.
Yeah. Dumb change. Undoing.
|
(!) A patch to the testing environment has been detected. |
2 similar comments
|
(!) A patch to the testing environment has been detected. |
|
(!) A patch to the testing environment has been detected. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism when dependencies allow. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> (cherry picked from commit 4d689f0) Change-Id: Id603f608575665c5f042dd83605a2bc6672525bc
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.