Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-24150 Allow module tests run in parallel #1464

Merged
merged 2 commits into from
Apr 9, 2020
Merged

HBASE-24150 Allow module tests run in parallel #1464

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3a52ca
HBASE-24150 Allow module tests run in parallel
saintstack Apr 9, 2020
260fbf6
Address feedback from Nick
saintstack Apr 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion dev-support/hbase-personality.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,10 @@ function personality_modules

clear_personality_queue

extra="-DHBasePatchProcess"
# At a few points, hbase modules can run their test suites in parallel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not just parallel test suites, but everything. With this change, maven will run up to 2 concurrent modules when the dependency graph permits. I imaging this will be a nice general benefit.

One thing to note though is that some plugins don't support multithreading. IIRC, surefire-reports was one such plugin, and would print warnings in the build. Looks like this patch didn't actually trigger any maven invocation, so I cannot point you at the message. A simple mvn clean package -DskipTests should be enough to get the message printed.

Since we use Jenkins to aggregate test results, I suggest we disable/remove surefire-reports (this was something I discussed with @busbey sometime back, but neither of us have gotten to it).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me change the comment.

I tried running the mvn clean ... above to find the message you refer to but too blind to see it. Let me add change in pom so its generated here.

On disabling the surefire reporting, you want that in this commit or on a follow-on? Is it this you are referring too?

maven-surefire-report-plugin

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's the plugin. Weirdly, now I try but I don't see the warning message. Hmm..

Running with -T1.0C I get a failure in

[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (check-aggregate-license) @ hbase-shaded-client ---                                                                                                           
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:                                                                                                                      
Couldn't evaluate condition: File license = new File("/Users/ndimiduk/repos/apache/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE");                                                                                                                                                                                                                                             
                    // Beanshell does not support try-with-resources,                                   
                    // so we must close this scanner manually                                                                                                                                                   
                    Scanner scanner = new Scanner(license);                                                                                                                                                     
                                                                                                                                                                                                                
                    while (scanner.hasNextLine()) {                                                                                                                                                             
                      if (scanner.nextLine().startsWith("ERROR:")) {                                                                                                                                            
                        scanner.close();                                                                                                                                                                        
                        return false;                                                                                                                                                                           
                      }                                                                                                                                                                                         
                    }                                                                                                                                                                                           
                    scanner.close();                                                                                                                                                                            
                    return true;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I see those WARNINGs locally too. Another issue?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it's not just a warning, it breaks the build. Yes another issue, but if it consistently breaks runs using -T, it would need to be a blocker of this one.

# Let it happen. Means we'll use more CPU but should be for short bursts.
# https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
extra="--threads=2 -DHBasePatchProcess"
if [[ "${PATCH_BRANCH}" = branch-1* ]]; then
extra="${extra} -Dhttps.protocols=TLSv1.2"
fi
Expand Down
4 changes: 2 additions & 2 deletions dev-support/hbase_nightly_source-artifact.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ fi
# See http://hbase.apache.org/book.html#maven.release

echo "Maven details, in case our JDK doesn't match expectations:"
mvn --version --offline | tee "${working_dir}/maven_version"
mvn --threads=2 --version --offline | tee "${working_dir}/maven_version"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no value in --threads for grabbing the version info, is there?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Dumb change. Undoing.


echo "Do a clean building of the source artifact using code in ${component_dir}"
cd "${component_dir}"
Expand Down Expand Up @@ -172,7 +172,7 @@ fi

cd "${unpack_dir}"
echo "Follow the ref guide section on making a RC: Step 8 Build the binary tarball."
if mvn -DskipTests -Prelease --batch-mode -Dmaven.repo.local="${m2_tarbuild}" clean install \
if mvn --threads=2 -DskipTests -Prelease --batch-mode -Dmaven.repo.local="${m2_tarbuild}" clean install \
assembly:single >"${working_dir}/srctarball_install.log" 2>&1; then
for artifact in "${unpack_dir}"/hbase-assembly/target/hbase-*-bin.tar.gz; do
if [ -f "${artifact}" ]; then
Expand Down