NIFIREG-186: Adding Ranger authorizer #131

ijokarumawak · 2018-07-20T07:02:55Z

Renger Authorizer is deployed as Registry extension.
Added /config REST endpoint to expose Registry configuration for UI to
determine if user, group and policies can be edited.

- Renger Authorizer is deployed as Registry extension. - Added /config REST endpoint to expose Registry configuration for UI to determine if user, group and policies can be edited.

scottyaslan · 2018-07-20T21:50:27Z

Reviewing...

scottyaslan · 2018-07-20T22:20:40Z

nifi-registry-web-ui/src/main/webapp/services/nf-registry.service.js

    };
+    nfRegistryApi.getRegistryConfig().subscribe(function (registryConfig) {


Please move this call to getRegistryConfig to https://github.com/apache/nifi-registry/blob/master/nifi-registry-web-ui/src/main/webapp/nf-registry.js#L56

kevdoran

Thanks for the contribution, @ijokarumawak! Overall, the code changes look good and follow the correct implementation pattern of the relevant NiFi Registry authorization APIs. I had a few inline comments on some minor issues, and here are a couple other considerations:

Building with -Pcontrib-checkturns up some check-style issues.
How do we envision distributing the Ranger functionality? Currently, in your branch, it looks like it is being built as a separate jar that needs to be added to a NiFi Registry /lib directory, is that correct? My only concern is that we now have an additional convenience binary to distribute. That might be unavoidable and the best thing to do, but I'm wondering if this would be better to bundle with NiFi Registry by default? (It would still require configuration in order to enable). Does it significantly increase the size of the distribution, or pull in dependencies we want to avoid in the main distribution?

Also, is there a certain build or version of Ranger that is required for this integration, or any special instructions for configuring things on the Ranger end?

kevdoran · 2018-07-23T16:16:46Z

nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ConfigResource.java

+            extensions = {
+                    @Extension(name = "access-policy", properties = {
+                            @ExtensionProperty(name = "action", value = "read"),
+                            @ExtensionProperty(name = "resource", value = "/config") })


This is minor as it only affects REST API documentation, but /config is not an authorizable ResourceType. I would change this line to include the two resource paths that are being used to authorize the request, so that those are reflected in the documentation:

@ExtensionProperty(name = "resource", value = "/policies,/tenants") })

kevdoran · 2018-07-23T16:19:17Z

nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/ConfigResource.java

+                            @ExtensionProperty(name = "resource", value = "/config") })
+            }
+    )
+    @ApiResponses({ @ApiResponse(code = 401, message = HttpStatusMessages.MESSAGE_401) })


Again, just a documentation issue, but as the body of the method includes an authorization check, this should include the possibility of a 403 response. That is:

@ApiResponses({ @ApiResponse(code = 401, message = HttpStatusMessages.MESSAGE_401), @ApiResponse(code = 403, message = HttpStatusMessages.MESSAGE_403) })

kevdoran · 2018-07-23T16:48:33Z

nifi-registry-ranger/src/test/resources/ranger/ranger-nifi-registry-security.xml

+	<property>
+		<name>ranger.plugin.nifi-registry.policy.rest.url</name>
+		<!-- TODO: Back to localhost -->
+		<value>http://192.168.99.100:6080</value>


Does the TODO comment here apply to this PR?

- Added 'include-ranger' maven build profile and refactored project structure to control ranger extension build. - Addressed checkstyle issues.

ijokarumawak · 2018-08-01T05:47:33Z

Thanks @scottyaslan @kevdoran for reviewing this! I've incorporated your feedback. Here are the answers to Kevin's previous questions:

How do we envision distributing the Ranger functionality?
I think we should follow how NiFi does with its Ranger component. I added 'include-ranger' build profile to bundle the extension into NiFi Registry binary. Since the extension is large, 50MB, I wouldn't recommend bundle it with Registry release by default.

If needed, anyone can build the extension easily by executing:

$ mvn clean install -Pcontrib-check -Pinclude-ranger -f nifi-registry-extensions/nifi-registry-ranger-extension

Then extract the built zip file 'nifi-registry-extensions/nifi-registry-ranger-extension/target/nifi-registry-ranger-extension-0.3.0-SNAPSHOT-bin.zip' into some directory that NiFi Registry can access.
Specify the directory from nifi-registry.properties:

nifi.registry.extension.dir.ranger=./ext/ranger/lib

We should document these somewhere though..

Also, is there a certain build or version of Ranger that is required for this integration, or any special instructions for configuring things on the Ranger end?

You need to apply a patch to Ranger since the NiFi Registry service is not merged into Ranger yet.
Please check https://issues.apache.org/jira/browse/RANGER-2157 to get the patch, or you can use my ranger branch. https://github.com/ijokarumawak/ranger/tree/nifi-registry

kevdoran · 2018-08-01T14:34:04Z

Thanks @ijokarumawak! I'll finish up my review based on your latest changes & comments.

- Added README.md to illustrate how to use this extension. - Added default configuration files.

ijokarumawak · 2018-08-07T06:39:27Z

nifi-registry-extensions/nifi-registry-ranger-extension/README.md

+
+## Prerequisites
+
+* Apache Ranger 1.2.0 or later is needed.


This sentence may need to be updated once the improvement at Ranger project is committed.

ijokarumawak · 2018-08-07T06:40:52Z

@kevdoran I've added a minimal documentation for users to use the Ranger extension. I hope it provides enough information. Thanks!

kevdoran

@ijokarumawak Thank you for the additions! The documentation looks good to me.

I wanted to provide you an update with where I am at reviewing this PR. I was able to setup a docker environment with NiFi Registry built from this PR branch and Ranger built from your PR branch to that project. They were able to authenticate to each other using mutual TLS authentication, and I was able to create policies in Ranger for my NiFi Registry users. I confirmed it works for a variety of user types (LDAP, local/file, etc.) So all that functionality looked good!

It did take me a while to configure Ranger, but that is because I'm completely new to building and configuring that from scratch. I think it would be easy for an experienced Ranger user/admin, and it all makes sense to me know that I've done it and understand how it is working.

Here are the scenarios I have not had a chance to test yet:

Kerberos-based authentication for Registry<->Ranger communication. I saw there are new properties for Kerberos Service principal/keytab location and a 'Ranger Kerberos Enabled' property for the Authorizer that toggles using that keytab to authenticate to Ranger.
HDFS-based Ranger auditing. From what I understand, the ranger library that performs auditing requires additional libraries in order to send audit records to HDFS. Do you know anything about this or have steps to test different auditing configurations? I plan to research it a bit and figure out if anything additional is needed for that on the Registry side. If additional libraries are needed, it may just be a documentation effort.

Can you think of anything else you would like me to review or verify regarding different configuration combinations?

Like I said, so far everything looks good. I just haven't verified the functionality that requires a special test environment (HDFS, KDC server, etc). If this is time sensitive, or if the Ranger project would prefer this PR be merged before merging your PR there, I would be happy to merge this as-is based on the functionality I have verified while I continue to test more scenarios, and if I find anything in further testing we could open a ticket for follow on work/improvements. Or, if there is no rush to merge this, we can leave it open and I'll continue to working on testing those other combinations. Let me know what you prefer.

ijokarumawak · 2018-08-11T08:48:00Z

@kevdoran Thanks for your effort on reviewing this. Glad to hear the Ranger authorization works in your environment, too. The two things Kerberos auth against Ranger and HDFS auditing, I haven't confirmed those either. That would be really helpful if you can test those, but I will try to do those myself, too.

I just added LICENSE and NOTICE for the extension package just in case so that we can distribute the assembled Ranger extension zip file if we want.

The improvement at Ranger side is under review, but not merged yet. I think we can wait until Ranger side gets merged. In the mean while, let's confirm it works with Kerberos and HDF audit.

kevdoran · 2018-08-11T10:37:11Z

Good idea regarding the L&N.

The improvement at Ranger side is under review, but not merged yet. I think we can wait until Ranger side gets merged. In the mean while, let's confirm it works with Kerberos and HDF audit.

Sounds good. I’ll have limited availability for the next week or so, but will verify those configurations as soon as possible. Thanks!

…onflict.

ijokarumawak · 2018-08-14T08:36:51Z

Note of the last commit. While I was deploying a different Ranger and Registry environment to test HDFS and Kerberos, I encountered a javax.ws.rs version conflict issue between jersey-bundle and nifi-registry. The jersey-bundle-1.19.3 used by Ranger common contains javax.ws.rs 1.x while NiFi Registry uses rs 2.1 api. This causes following exception occasionally:

Caused by: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/home/koji/nifi-registry-0.3.0-SNAPSHOT/work/jetty/nifi-registry-web-api-0.3.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.ws.rs-api-2.1.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/home/koji/nifi-registry-0.3.0-SNAPSHOT/./ext/ranger/lib/jersey-bundle-1.19.3.jar!/javax/ws/rs/ext/RuntimeDelegate.class
        at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:116) ~[na:na]
        at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) ~[na:na]
        at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44) ~[na:na]
        at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65) ~[na:na]
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182) ~[na:na]
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175) ~[na:na]
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162) ~[na:na]
        at com.sun.jersey.api.client.Client.init(Client.java:343) ~[na:na]
        at com.sun.jersey.api.client.Client.access$000(Client.java:119) ~[na:na]
        at com.sun.jersey.api.client.Client$1.f(Client.java:192) ~[na:na]
        at com.sun.jersey.api.client.Client$1.f(Client.java:188) ~[na:na]
        at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193) ~[na:na]
        at com.sun.jersey.api.client.Client.<init>(Client.java:188) ~[na:na]
        at com.sun.jersey.api.client.Client.<init>(Client.java:171) ~[na:na]
        at com.sun.jersey.api.client.Client.create(Client.java:683) ~[na:na]
        at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:211) ~[na:na]
        at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176) ~[na:na]
        at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156) ~[na:na]
        at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:275) ~[na:na]
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126) ~[na:na]
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264) ~[na:na]
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202) ~[na:na]
        at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149) ~[na:na]
        at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:150) ~[na:na]
        at org.apache.nifi.registry.ranger.RangerAuthorizer.onConfigured(RangerAuthorizer.java:165) ~[na:na]
        ... 69 common frames omitted

To avoid this, I've added mvn shade plugin to remove javax.ws.rs package from the jersey-bundle dependency.

- Write more docs and provide more example configurations. - Updated hadoop version to 3.0.0 - Added ExtensionCloseable to use extension class loader when configuring authorizer. Without this, Hadoop Configuration class uses WebApp class loader that is set to current thread context class loader which does not have extension classes. - Refactored anonymous inner classes at AuthorizerFactory to expose underlying authorizer instance, to use its extension class loader. - Confirmed NiFi Registry can: - download policies from Kerbelized Ranger - send audit logs to Kerbelized Solr - send audit logs to Kerbelized HDFS

ijokarumawak · 2018-08-20T05:08:18Z

@kevdoran I've finished writing L&N and test with a Kerberized Ranger, HDFS and Solr. I hope this PR is ready to get merged into NiFi Registry project. Please let me know if you have any concern. Let's wait for Ranger side of things to be ready. I will ping Ranger community once more. Thanks!

kevdoran · 2018-08-20T13:23:39Z

@ijokarumawak Thanks for the update.

I'll review the most recent commits sometime over the next few days and test with those additional configurations. I'll watch the Ranger PR that is still under review, and help get this PR merged once that is finalized. Let me know if I can help to get either pull request finalized, reviewed, or tested.

kevdoran · 2018-09-18T16:41:05Z

nifi-registry-extensions/nifi-registry-ranger/nifi-registry-ranger-assembly/README.md

+
+## Prerequisites
+
+* Apache Ranger 1.2.0 or later is needed.


Is this correct? Based on the changes submitted to Apache Ranger that will add support for this, it looks like it will require Apache Ranger >= 2.0.0, correct?

Good point. I think that is a moving part. Ranger seems to release 1.2.0 soon, but the changes are not merged yet. I'll check. https://issues.apache.org/jira/browse/RANGER-2228

Ok, looks like this is good for Ranger 1.2.0 and 2.0.0. I'll merge it as-is then, thanks!

kevdoran

+1, merging to master. Great work @ijokarumawak, both on the Ranger and Registry side, this is a nice addition!

NIFIREG-186: Adding Ranger authorizer

11e4734

- Renger Authorizer is deployed as Registry extension. - Added /config REST endpoint to expose Registry configuration for UI to determine if user, group and policies can be edited.

scottyaslan reviewed Jul 20, 2018

View reviewed changes

kevdoran reviewed Jul 23, 2018

View reviewed changes

Incorporated review comments

3f7eb37

- Added 'include-ranger' maven build profile and refactored project structure to control ranger extension build. - Addressed checkstyle issues.

Added minimum docs to use Ranger extension.

29239be

- Added README.md to illustrate how to use this extension. - Added default configuration files.

ijokarumawak force-pushed the nifireg-186 branch from 206d2e9 to 29239be Compare August 7, 2018 06:35

ijokarumawak commented Aug 7, 2018

View reviewed changes

kevdoran approved these changes Aug 10, 2018

View reviewed changes

Added LICENSE and NOTICE.

349485d

Remove javax.ws.rs package in jersey-bundle jar to avoid rs version c…

7057ff2

…onflict.

ijokarumawak added 3 commits August 17, 2018 11:18

Added example conf to audit to Kerberized Solr.

21424f1

Refactored project structures and updated L&N.

2118d3d

kevdoran reviewed Sep 18, 2018

View reviewed changes

kevdoran approved these changes Sep 20, 2018

View reviewed changes

asfgit closed this in e1bd6e2 Sep 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFIREG-186: Adding Ranger authorizer #131

NIFIREG-186: Adding Ranger authorizer #131

ijokarumawak commented Jul 20, 2018

scottyaslan commented Jul 20, 2018

scottyaslan Jul 20, 2018

kevdoran left a comment •

edited

kevdoran Jul 23, 2018

kevdoran Jul 23, 2018

kevdoran Jul 23, 2018

ijokarumawak commented Aug 1, 2018

kevdoran commented Aug 1, 2018

ijokarumawak Aug 7, 2018

ijokarumawak commented Aug 7, 2018

kevdoran left a comment •

edited

ijokarumawak commented Aug 11, 2018

kevdoran commented Aug 11, 2018 •

edited

ijokarumawak commented Aug 14, 2018

ijokarumawak commented Aug 20, 2018

kevdoran commented Aug 20, 2018

kevdoran Sep 18, 2018 •

edited

ijokarumawak Sep 18, 2018

kevdoran Sep 20, 2018

kevdoran left a comment

		};
		nfRegistryApi.getRegistryConfig().subscribe(function (registryConfig) {

NIFIREG-186: Adding Ranger authorizer #131

NIFIREG-186: Adding Ranger authorizer #131

Conversation

ijokarumawak commented Jul 20, 2018

scottyaslan commented Jul 20, 2018

scottyaslan Jul 20, 2018

Choose a reason for hiding this comment

kevdoran left a comment • edited

Choose a reason for hiding this comment

kevdoran Jul 23, 2018

Choose a reason for hiding this comment

kevdoran Jul 23, 2018

Choose a reason for hiding this comment

kevdoran Jul 23, 2018

Choose a reason for hiding this comment

ijokarumawak commented Aug 1, 2018

kevdoran commented Aug 1, 2018

ijokarumawak Aug 7, 2018

Choose a reason for hiding this comment

ijokarumawak commented Aug 7, 2018

kevdoran left a comment • edited

Choose a reason for hiding this comment

ijokarumawak commented Aug 11, 2018

kevdoran commented Aug 11, 2018 • edited

ijokarumawak commented Aug 14, 2018

ijokarumawak commented Aug 20, 2018

kevdoran commented Aug 20, 2018

kevdoran Sep 18, 2018 • edited

Choose a reason for hiding this comment

ijokarumawak Sep 18, 2018

Choose a reason for hiding this comment

kevdoran Sep 20, 2018

Choose a reason for hiding this comment

kevdoran left a comment

Choose a reason for hiding this comment

kevdoran left a comment •

edited

kevdoran left a comment •

edited

kevdoran commented Aug 11, 2018 •

edited

kevdoran Sep 18, 2018 •

edited