Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

Merged
merged 1 commit into from
May 26, 2022
Merged

HBASE-27061 two phase bulkload is broken when SFT is in use. #4465

merged 1 commit into from
May 26, 2022

Conversation

ss77892
Copy link
Contributor

@ss77892 ss77892 commented May 26, 2022 •
edited
Loading

o reproduce the bug I've used hbase with hbase root dir pointing to s3 bucket and WAL dir pointing to HDFS. 2 phase bulkload was used:
Phase1 (Completed successfully):
hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=',' -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,name,department employees /tmp/test.csv

Phase2 (Failed with IllegalArgumentException on the RS side):
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles output employees

With the provided changes Phase2 was completed successfully.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 2m 45s master passed
+1 💚 compile 0m 48s master passed
+1 💚 shadedjars 3m 42s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 29s master passed
_ Patch Compile Tests _
-1 ❌ mvninstall 1m 24s root in the patch failed.
-1 ❌ compile 0m 29s hbase-server in the patch failed.
-0 ⚠️ javac 0m 29s hbase-server in the patch failed.
-1 ❌ shadedjars 2m 44s patch has 16 errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 26s the patch passed
_ Other Tests _
-1 ❌ unit 0m 29s hbase-server in the patch failed.
16m 23s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4465
Optional Tests javac javadoc unit shadedjars compile
uname Linux 490ed89d2197 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-11.0.10+9
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/testReport/
Max. process+thread count 70 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 3m 33s master passed
+1 💚 compile 2m 10s master passed
+1 💚 checkstyle 0m 31s master passed
+1 💚 spotless 0m 46s branch has no errors when running spotless:check.
+1 💚 spotbugs 1m 12s master passed
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 59s root in the patch failed.
-1 ❌ compile 0m 25s hbase-server in the patch failed.
-0 ⚠️ javac 0m 25s hbase-server in the patch failed.
+1 💚 checkstyle 0m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ hadoopcheck 1m 3s The patch causes 16 errors with Hadoop v3.1.2.
-1 ❌ hadoopcheck 2m 18s The patch causes 16 errors with Hadoop v3.2.2.
-1 ❌ hadoopcheck 3m 30s The patch causes 16 errors with Hadoop v3.3.1.
+1 💚 spotless 0m 34s patch has no errors when running spotless:check.
-1 ❌ spotbugs 0m 20s hbase-server in the patch failed.
_ Other Tests _
+1 💚 asflicense 0m 8s The patch does not generate ASF License warnings.
16m 25s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4465
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux f2e3be0c3b5c 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.2.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt
Max. process+thread count 60 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 3s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 11s master passed
+1 💚 compile 0m 42s master passed
+1 💚 shadedjars 4m 25s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 34s master passed
_ Patch Compile Tests _
-1 ❌ mvninstall 1m 21s root in the patch failed.
-1 ❌ compile 0m 29s hbase-server in the patch failed.
-0 ⚠️ javac 0m 29s hbase-server in the patch failed.
-1 ❌ shadedjars 2m 54s patch has 16 errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 21s the patch passed
_ Other Tests _
-1 ❌ unit 0m 22s hbase-server in the patch failed.
17m 21s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4465
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8c1cbcbd5a7a 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/testReport/
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 2m 4s master passed
+1 💚 compile 2m 9s master passed
+1 💚 checkstyle 0m 26s master passed
+1 💚 spotless 0m 37s branch has no errors when running spotless:check.
+1 💚 spotbugs 1m 9s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 7s the patch passed
+1 💚 compile 2m 7s the patch passed
+1 💚 javac 2m 7s the patch passed
+1 💚 checkstyle 0m 27s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 11m 24s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotless 0m 37s patch has no errors when running spotless:check.
+1 💚 spotbugs 1m 16s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 7s The patch does not generate ASF License warnings.
30m 6s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4465
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux c551550358e1 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 43s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 2m 11s master passed
+1 💚 compile 0m 34s master passed
+1 💚 shadedjars 3m 51s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 21s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 10s the patch passed
+1 💚 compile 0m 35s the patch passed
+1 💚 javac 0m 35s the patch passed
+1 💚 shadedjars 3m 56s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 21s the patch passed
_ Other Tests _
+1 💚 unit 196m 26s hbase-server in the patch passed.
213m 14s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4465
Optional Tests javac javadoc unit shadedjars compile
uname Linux 5d338ca60ce8 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/testReport/
Max. process+thread count 2636 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 11s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 2m 34s master passed
+1 💚 compile 0m 46s master passed
+1 💚 shadedjars 3m 41s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 2m 32s the patch passed
+1 💚 compile 0m 46s the patch passed
+1 💚 javac 0m 46s the patch passed
+1 💚 shadedjars 3m 40s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 27s the patch passed
_ Other Tests _
+1 💚 unit 201m 35s hbase-server in the patch passed.
219m 5s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4465
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8cd380abd9e9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 286f6c5
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/testReport/
Max. process+thread count 2460 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4465/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil wchevreuil merged commit fdf054d into apache:master May 26, 2022
wchevreuil pushed a commit that referenced this pull request May 26, 2022
@ss77892 @wchevreuil
HBASE-27061 two phase bulkload is broken when SFT is in use. (#4465)
50dd961 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit that referenced this pull request May 26, 2022
@ss77892 @wchevreuil
HBASE-27061 two phase bulkload is broken when SFT is in use. (#4465)
c9fc396 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wenwj0 pushed a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022
@ss77892 @wenwj0
HBASE-27061 two phase bulkload is broken when SFT is in use. (apache#…
ae494c1 
…4465)

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wenwj0 added a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022
@wenwj0
Revert "HBASE-27061 two phase bulkload is broken when SFT is in use. (a…
11967b8 
…pache#4465)"

This reverts commit ae494c1.
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@ss77892
HBASE-27061 two phase bulkload is broken when SFT is in use. (apache#…
811ed17 
…4465)

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Change-Id: I21f336eed6a597ff84c7014db32d5d052681b652
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wchevreuil wchevreuil wchevreuil approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@ss77892 @Apache-HBase @wchevreuil