Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-21995 Add a coprocessor to set HDFS ACL for hbase granted user #163

Merged
merged 1 commit into from Jun 24, 2019

Conversation

@mymeiyi
Copy link
Contributor

mymeiyi commented Apr 18, 2019

To make hbase granted user have the access to scan table snapshots, use HDFS ACLs to set user 'access r-x' or 'default r-x' ACLs over hfiles.
The basic implementation is:

  1. For public directories such as 'data' and 'archive', set other users' permission to '--x' to make everyone have the permission to access the directory.
  2. For namespace or table directories such as 'data/ns/table', 'archive/data/ns/table' and '.hbase-snapshot/snapshotName', set user 'r-x' acl and default 'r-x' acl when grant, revoke, snapshot.
    The feature is configurable because it's implemented in a master coprocessor.
@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented Apr 18, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 394 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 mvninstall 302 master passed
+1 compile 55 master passed
+1 checkstyle 72 master passed
+1 shadedjars 281 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 215 master passed
+1 javadoc 34 master passed
_ Patch Compile Tests _
+1 mvninstall 258 the patch passed
+1 compile 53 the patch passed
+1 javac 53 the patch passed
-1 checkstyle 67 hbase-server: The patch generated 3 new + 51 unchanged - 2 fixed = 54 total (was 53)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 269 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 554 Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.
+1 findbugs 213 the patch passed
+1 javadoc 34 the patch passed
_ Other Tests _
-1 unit 12876 hbase-server in the patch failed.
+1 asflicense 28 The patch does not generate ASF License warnings.
15781
Reason Tests
Failed junit tests hadoop.hbase.client.TestFromClientSide
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/1/artifact/out/Dockerfile
GITHUB PR #163
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 495d4df97ab4 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 428afa9
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/1/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/1/testReport/
Max. process+thread count 4769 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@@ -314,12 +316,24 @@ private FSDataInputStream tryOpen() throws IOException {
return(in);
} catch (FileNotFoundException e) {
// Try another file location
} catch (AccessControlException e) {

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

Here, I prefer to simplify the logic as a small method:

  1. remember the thrown exception as e;
  2. if notfound or accessControl exception, continue to try another file;
  3. if still not find an right file. then throw the e.
    Please consider this.

This comment has been minimized.

Copy link
@openinx

openinx Jun 4, 2019

Member

Please abstract all the exception handling logic as method named handleException ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi Jun 12, 2019

Author Contributor

done

for (int i = 0; i < locations.length; ++i) {
try {
return fs.getFileStatus(locations[i]);
} catch (FileNotFoundException e) {
// Try another file location
} catch (AccessControlException e) {

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

Same question above.

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi Jun 12, 2019

Author Contributor

done

*/
@CoreCoprocessor
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
public class HDFSAclController implements MasterCoprocessor, MasterObserver {

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

Not a good class name, the class want to sync file acl between HBase and HDFS ? and mostly for those directories when scanning snapshot ? we don't consider those directories which is unrelated to snapshot, such as WAL, oldWals etc... Please consider another name.

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi May 15, 2019

Author Contributor

How about "SnapshotScannerHDFSAclController"?

}

@Override
public void preMasterInitialization(final ObserverContext<MasterCoprocessorEnvironment> c)

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

the final can remove now in jdk8 ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi Jun 12, 2019

Author Contributor

done

masterServices = ((HasMasterServices) mEnv).getMasterServices();
}
if (masterServices == null) {
throw new RuntimeException("master services can not be null");

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

IllegalArgumentException ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi Jun 12, 2019

Author Contributor

done

if (!fs.exists(path)) {
fs.mkdirs(path);
}
fs.setPermission(path, ACL_ENABLE_PUBLIC_HFILE_PERMISSION);

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

This acl need also to be configurable ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi Jun 12, 2019

Author Contributor

done

ColumnFamilyDescriptorBuilder.newBuilder(HDFSAclStorage.HDFS_ACL_FAMILY).build());
admin.modifyTable(builder.build());
}
}

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

What if the hbase:acl does not exist ? should throw an exception ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi May 15, 2019

Author Contributor

This coprocessor should be configured after the AccessController, if hbase:acl table does not exist, the AccessController will not work incorrectly firstly?
Let me add some logs and throw an TableNotFoundException here.

try (Admin admin = ctx.getEnvironment().getConnection().getAdmin()) {
if (admin.tableExists(PermissionStorage.ACL_TABLE_NAME)) {
// check if hbase:acl table has 'm' CF
TableDescriptor tableDescriptor = admin.getDescriptor(PermissionStorage.ACL_TABLE_NAME);

This comment has been minimized.

Copy link
@openinx

openinx Apr 19, 2019

Member

Would the newly introduced CF impact the original AccessController ?

This comment has been minimized.

Copy link
@mymeiyi

mymeiyi May 15, 2019

Author Contributor

The new CF is only used in this CP, it records if the hbase read permission is synchronized to related hfile.
This flag has two usages:

  1. check if we need to remove hdfs acls for a grant without READ permission;
  2. skip some hdfs acl sync because it may be already added.
@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented Apr 21, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 38 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 mvninstall 310 master passed
+1 compile 67 master passed
+1 checkstyle 85 master passed
+1 shadedjars 337 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 287 master passed
+1 javadoc 41 master passed
_ Patch Compile Tests _
+1 mvninstall 302 the patch passed
+1 compile 68 the patch passed
+1 javac 68 the patch passed
-1 checkstyle 83 hbase-server: The patch generated 3 new + 51 unchanged - 2 fixed = 54 total (was 53)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 339 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 621 Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.
+1 findbugs 298 the patch passed
+1 javadoc 40 the patch passed
_ Other Tests _
+1 unit 14069 hbase-server in the patch passed.
+1 asflicense 31 The patch does not generate ASF License warnings.
17083
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/2/artifact/out/Dockerfile
GITHUB PR #163
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 3763afd5e56d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / f30d6c9
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/2/artifact/out/diff-checkstyle-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/2/testReport/
Max. process+thread count 4655 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented Apr 30, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 7 #163 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
Subsystem Report/Notes
GITHUB PR #163
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented May 7, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 58 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 mvninstall 331 master passed
+1 compile 58 master passed
+1 checkstyle 72 master passed
+1 shadedjars 289 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 228 master passed
+1 javadoc 36 master passed
_ Patch Compile Tests _
+1 mvninstall 259 the patch passed
+1 compile 60 the patch passed
+1 javac 60 the patch passed
-1 checkstyle 76 hbase-server: The patch generated 4 new + 51 unchanged - 2 fixed = 55 total (was 53)
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 275 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 601 Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.
-1 findbugs 239 hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 javadoc 34 the patch passed
_ Other Tests _
-1 unit 20409 hbase-server in the patch failed.
+1 asflicense 31 The patch does not generate ASF License warnings.
23451
Reason Tests
FindBugs module:hbase-server
Exception is caught when Exception is not thrown in org.apache.hadoop.hbase.security.access.HDFSAclHelper.grant(byte[], Set) At HDFSAclHelper.java:is not thrown in org.apache.hadoop.hbase.security.access.HDFSAclHelper.grant(byte[], Set) At HDFSAclHelper.java:[line 152]
Failed junit tests hadoop.hbase.master.TestMasterMetricsWrapper
hadoop.hbase.security.access.TestHDFSAclController
hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas
hadoop.hbase.client.TestHbck
hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.master.procedure.TestTruncateTableProcedure
hadoop.hbase.tool.TestLoadIncrementalHFiles
hadoop.hbase.master.TestAssignmentManagerMetrics
hadoop.hbase.client.TestAdmin1
hadoop.hbase.client.replication.TestReplicationAdminWithClusters
hadoop.hbase.master.procedure.TestProcedurePriority
hadoop.hbase.replication.TestReplicationSyncUpTool
hadoop.hbase.namespace.TestNamespaceAuditor
hadoop.hbase.replication.TestReplicationSmallTestsSync
hadoop.hbase.client.TestFromClientSide
hadoop.hbase.master.TestSplitWALManager
hadoop.hbase.master.procedure.TestSCPWithoutZKCoordinated
hadoop.hbase.client.TestFromClientSideWithCoprocessor
hadoop.hbase.util.TestFromClientSide3WoUnsafe
hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory
hadoop.hbase.tool.TestSecureLoadIncrementalHFiles
hadoop.hbase.client.TestSnapshotTemporaryDirectory
hadoop.hbase.replication.TestReplicationSmallTests
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/artifact/out/Dockerfile
GITHUB PR #163
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux b3efbe6fa87c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 67c937f
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/artifact/out/diff-checkstyle-hbase-server.txt
findbugs https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/artifact/out/new-findbugs-hbase-server.html
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/testReport/
Max. process+thread count 5240 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/4/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@openinx

This comment has been minimized.

Copy link
Member

openinx commented May 7, 2019

@mymeiyi Please check the failed UT.

@Apache9

This comment has been minimized.

Copy link
Contributor

Apache9 commented May 14, 2019

Any updates here?

@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented May 15, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 10 #163 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
Subsystem Report/Notes
GITHUB PR #163
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/5/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented May 16, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 8 #163 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
Subsystem Report/Notes
GITHUB PR #163
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/6/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

This comment has been minimized.

Copy link

Apache-HBase commented May 30, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 6 #163 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
Subsystem Report/Notes
GITHUB PR #163
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-163/7/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.