Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-26168 Backport HBASE-25811 "The client integration test is fail… #3634

Merged
merged 1 commit into from
Aug 27, 2021
Merged

HBASE-26168 Backport HBASE-25811 "The client integration test is fail… #3634

merged 1 commit into from
Aug 27, 2021

Conversation

taklwu
Copy link
Contributor

@taklwu taklwu commented Aug 26, 2021

…ing after HBASE-22120 merged" into branch-2

move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

original commit 8d2a0ef

@Apache9 @taklwu
HBASE-26168 Backport HBASE-25811 "The client integration test is fail…
723831a 
…ing after HBASE-22120 merged" into branch-2

move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

original commit 8d2a0ef
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ HBASE-25853 Compile Tests _
+0 🆗 mvndep 0m 17s Maven dependency ordering for branch
+1 💚 mvninstall 3m 42s HBASE-25853 passed
+1 💚 compile 1m 14s HBASE-25853 passed
+1 💚 checkstyle 0m 33s HBASE-25853 passed
+1 💚 spotbugs 0m 47s HBASE-25853 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 3m 21s the patch passed
+1 💚 compile 1m 13s the patch passed
+1 💚 javac 1m 13s the patch passed
+1 💚 checkstyle 0m 30s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 3s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 11m 59s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 spotbugs 0m 55s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 26s The patch does not generate ASF License warnings.
33m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #3634
Optional Tests dupname asflicense javac xml spotbugs hadoopcheck hbaseanti checkstyle compile
uname Linux 5d75ec136fa9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision HBASE-25853 / d807632
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 96 (vs. ulimit of 12500)
modules C: hbase-mapreduce hbase-assembly U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@taklwu
Copy link
Contributor Author

taklwu commented Aug 26, 2021

this commit is required for the completion of HBASE-25853, otherwise the import TSV nightly job with shaded client artifact fail with NoClassDefFoundError exception. and I tested the nightly tests locally with the following steps (with outputs)

# first maven full build 
HBASE_SOURCE=/Users/taklwu/workspace/hbase2
cd $HBASE_SOURCE
mvn clean install -DskipTests -Prelease && mvn install -DskipTests site assembly:single -Prelease

# unzip the assembly and hbase-client
cd ~/hbase-nightly         
install_artifact=$(ls -1 ${HBASE_SOURCE}/hbase-assembly/target/hbase-*-bin.tar.gz | sort | head -n 1)
mkdir hbase-install   
tar --strip-component=1 -xzf "${install_artifact}" -C "hbase-install"
client_artifact=$(ls -1 ${HBASE_SOURCE}/hbase-assembly/target/hbase-*-bin.tar.gz | sort | tail -n 1)
mkdir hbase-client   
tar --strip-component=1 -xzf "${client_artifact}" -C "hbase-client"

# run the nightly
rm -rf output-integration/hadoop-2/*
bash $HBASE_SOURCE/dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process --working-dir output-integration/hadoop-2 --hbase-client-install hbase-client hbase-install hadoop-2/bin/hadoop hadoop-2/share/hadoop/yarn/timelineservice hadoop-2/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.10.1-tests.jar hadoop-2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.1-tests.jar hadoop-2/bin/mapred | tee ~/hbase-nightly/sanity-check.log

Using HBase client-side artifact
You'll find logs and temp files in /Users/taklwu/hbase-nightly/output-integration/hadoop-2
Hadoop version information:
Hadoop 2.10.1
Subversion https://github.com/apache/hadoop -r 1827467c9a56f133025f28557bfc2c562d78e816
Compiled by centos on 2020-09-14T13:17Z
Compiled with protoc 2.5.0
From source with checksum 3114edef868f1f3824e7d0f68be03650
This command was run using /Users/taklwu/hbase-nightly/hadoop-2/share/hadoop/common/hadoop-common-2.10.1.jar
JAVA_HOME: /Users/taklwu/.sdkman/candidates/java/8.292.10.1-amzn
HBase version information:
HBase 2.5.0-SNAPSHOT
Source code repository git://21992.local/Users/taklwu/workspace/hbase2 revision=723831a9de938faf7c2a1b2e4047b68ed281af52
Compiled by taklwu on Thu Aug 26 14:29:11 PDT 2021
From source with checksum c90bbbd1a0293de7e3621195fad2d2e6ade1a2b82fdcb05ee7690e3fb8b338a743d7428dee3ef18b9c6b6d7de1ea979bc8073f5b9bb686b24fc48aa4e8b48e90
Writing out configuration for HBase.
Starting up Hadoop
	waiting for Hadoop to finish starting up.
	waiting for Hadoop to finish starting up.
	waiting for Hadoop to finish starting up.
Listing HDFS contents
Starting up HBase
running master, logging to /Users/taklwu/hbase-nightly/hbase-install/bin/../logs/hbase-taklwu-master-21992.local.out
	retry waiting for hbase to come up.
Setting up table 'test:example' with 1,000 regions
writing out example TSV to example.tsv
uploading example.tsv to HDFS
2021-08-26 14:59:41,249 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-08-26 14:59:42,671 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Importing TSV via shaded client artifact for HBase - MapReduce integration.
Verifying row count from import.
Hadoop client jars not given; getting them from 'hadoop classpath' for the example.
Building shaded client example.
Running shaded client example. It'll fetch the set of regions, round-trip them to a file in HDFS, then write them one-per-row into the test table.
Checking on results of example program.
2021-08-26 15:02:29,669 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Verifying row count from example.
Found 1051 rows, which is enough to cover 48 for import, 1000 example's use of user table regions, 2 for example's use of meta/namespace regions, and 1 for example's count record
Shutting down HBase
stopping hbase............
Shutdown: listing HDFS contents
Shutting down Hadoop

# check the command exit code and output of  mr-importtsv.err
% echo $?
0

% cat output-integration/hadoop-2/mr-importtsv.err
...
2021-08-26 15:00:00,482 INFO  [main] mapreduce.Job: Job job_local361623923_0001 completed successfully
2021-08-26 15:00:00,487 INFO  [main] mapreduce.Job: Counters: 21
	File System Counters
		FILE: Number of bytes read=47342109
		FILE: Number of bytes written=48290866
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=756
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=3
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Map-Reduce Framework
		Map input records=48
		Map output records=48
		Input split bytes=119
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=556793856
	ImportTsv
		Bad Lines=0
	File Input Format Counters
		Bytes Read=756
	File Output Format Counters
		Bytes Written=0
...

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 48s Docker mode activated.
-0 ⚠️ yetus 0m 8s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-25853 Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for branch
+1 💚 mvninstall 5m 51s HBASE-25853 passed
+1 💚 compile 1m 4s HBASE-25853 passed
+1 💚 shadedjars 10m 23s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 44s HBASE-25853 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 5m 18s the patch passed
+1 💚 compile 1m 11s the patch passed
+1 💚 javac 1m 11s the patch passed
+1 💚 shadedjars 9m 46s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 40s the patch passed
_ Other Tests _
+1 💚 unit 10m 33s hbase-mapreduce in the patch passed.
+1 💚 unit 0m 27s hbase-assembly in the patch passed.
49m 13s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #3634
Optional Tests javac javadoc unit shadedjars compile
uname Linux 0e7b949ce26a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision HBASE-25853 / d807632
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/testReport/
Max. process+thread count 4837 (vs. ulimit of 12500)
modules C: hbase-mapreduce hbase-assembly U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 40s Docker mode activated.
-0 ⚠️ yetus 0m 7s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-25853 Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for branch
+1 💚 mvninstall 4m 50s HBASE-25853 passed
+1 💚 compile 0m 59s HBASE-25853 passed
+1 💚 shadedjars 8m 27s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 40s HBASE-25853 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 17s Maven dependency ordering for patch
+1 💚 mvninstall 4m 44s the patch passed
+1 💚 compile 1m 2s the patch passed
+1 💚 javac 1m 2s the patch passed
+1 💚 shadedjars 8m 40s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 38s the patch passed
_ Other Tests _
+1 💚 unit 18m 39s hbase-mapreduce in the patch passed.
+1 💚 unit 0m 30s hbase-assembly in the patch passed.
53m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #3634
Optional Tests javac javadoc unit shadedjars compile
uname Linux b3f478963855 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision HBASE-25853 / d807632
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/testReport/
Max. process+thread count 2616 (vs. ulimit of 12500)
modules C: hbase-mapreduce hbase-assembly U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3634/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@taklwu taklwu merged commit fc779d1 into apache:HBASE-25853 Aug 27, 2021
taklwu added a commit that referenced this pull request Aug 27, 2021
@taklwu
HBASE-26168 Backport HBASE-25811 "The client integration test is fail…
791fd64 
…ing after HBASE-22120 merged" into branch-2 (#3634)

move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

original commit 8d2a0ef

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
taklwu added a commit that referenced this pull request Sep 1, 2021
@taklwu
HBASE-26168 Backport HBASE-25811 "The client integration test is fail…
625093f 
…ing after HBASE-22120 merged" into branch-2 (#3634)

move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

original commit 8d2a0ef

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@taklwu
HBASE-26168 Backport HBASE-25811 "The client integration test is fail…
b9bd4c0 
…ing after HBASE-22120 merged" into branch-2 (apache#3634)

move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

original commit 8d2a0ef

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>

(cherry picked from commit 625093f)
Ref: CDPD-32289
Reason: CVE / Security
Note: other than we have one line difference in TableMapReduceUtil.java, it's a clean backport from upstream
Change-Id: Ibec6c8828cec0ac8a48bfaff6e9627a07496b3d0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@petersomogyi petersomogyi petersomogyi approved these changes

@Apache9 Apache9 Apache9 approved these changes

@apurtell apurtell Awaiting requested review from apurtell

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@taklwu @Apache-HBase @petersomogyi @Apache9