Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-27630: hbase-spark bulkload stage directory limited to hdfs only #108

Merged
merged 1 commit into from
Feb 14, 2023

Conversation

ss77892
Copy link
Contributor

@ss77892 ss77892 commented Feb 9, 2023

The reason for the problem was the usage of the default FileSystem instances for creating and using staging stuff. Now we use the staging path to obtain the FileSystem instance, so it should work correctly for any different from the hdfs schema.
The test uses a local filesystem and doesn't load files to the table due to limitations:

  1. HBase minicluster uses hdfs only, and there is no way to specify a custom hbase.rootdir for that. So the load would not work because the staging dir is on a different filesystem
  2. I tried to get hadoop-aws mock implementation working for an external filesystem, but that would require making the tests for this module published first

Both those two bullets can be addressed in separate jiras if required.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 42s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 1m 36s master passed
+1 💚 compile 0m 38s master passed
+1 💚 scaladoc 0m 48s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 48s the patch passed
+1 💚 compile 0m 39s the patch passed
+1 💚 scalac 0m 39s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 scaladoc 0m 47s the patch passed
_ Other Tests _
+1 💚 unit 7m 35s hbase-spark in the patch passed.
14m 16s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-108/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #108
Optional Tests dupname scalac scaladoc unit compile
uname Linux 1c4312f80fa8 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / c221896
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-108/1/testReport/
Max. process+thread count 945 (vs. ulimit of 12500)
modules C: spark/hbase-spark U: spark/hbase-spark
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-108/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty
Copy link
Contributor

stoty commented Feb 14, 2023

LGTM

Copy link
Contributor

@stoty stoty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM, but I am not a committer.

@petersomogyi petersomogyi merged commit 672f3de into apache:master Feb 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants