[WIP] Presto ranger integration #244

cryptoe · 2019-02-15T16:49:43Z

Porting project to prestoSQL with review comments addressed.

I was planning to use testing https://github.com/prestosql/presto/blob/master/presto-main/src/main/java/io/prestosql/server/testing/TestingPrestoServer.java which uses presto-main dependency and internally brings jersey 2.xx jars. As ranger client uses jersey 1.xx jars, I was not able to write test cases for the same. Need pointers to solve this unit test cases issue.
@stagraqubole @dain @martint

cla-bot · 2019-02-15T16:49:45Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to cla@prestosql.io. If you are contributing on behalf of someone else (e.g., your employer), the individual CLA may not be sufficient and your employer may need the Corporate CLA signed.

shubhamtagra · 2019-02-18T04:13:16Z

As discussed over slack, we could use product-tests to test the integration.

cryptoe · 2019-03-06T15:47:00Z

@stagraqubole Added the product tests and documentation.

shubhamtagra · 2019-03-18T04:56:50Z

presto-ranger-plugin/src/main/java/io/prestosql/ranger/PrestoAuthorizer.java

+        return userGroups.getUserGroups(identity.getUser());
+    }
+
+    public String getRowLevelFilterExp(String catalogName, RangerPrestoResource resource, Identity identity)


The row filter methods should be removed, they arent being used right now.

column mask changes too

shubhamtagra · 2019-03-18T05:06:27Z

presto-ranger-plugin/src/main/java/io/prestosql/ranger/RangerSystemAccessControlFactory.java

+    @Override
+    public SystemAccessControl create(Map<String, String> config)
+    {
+        RangerConfiguration rangerConfig = RangerConfiguration.getInstance();


You should store away the pluginClassLoader in constructor and use it in create() method.

We can follow the example of ClassLoaderSafeConnectorMetadata to create a wrapper on top of RangerSystemAccessControl to use pluginClassloader for all SystemAccessControl functions.

shubhamtagra · 2019-03-18T05:14:38Z

The product tests are using file based policy client which is not used outside of tests. I was hoping we could get a container image of ranger, set it up with some preset policies and use the real world RestClient of Ranger for the tests. But wait on the comments from @martint, @dain or @electrum before you put in effort on this.

bolkedebruin · 2019-03-27T08:44:03Z

I have created a boilerplate plugin that works with recent Ranger. It is part of the Ranger source code instead of Presto, as Ranger as the infrastructure for creating these type of plugins. Will work on integrating it into Ranger itself.

https://github.com/bolkedebruin/ranger/tree/presto_plugin

bolkedebruin · 2019-04-04T20:19:15Z

@Phelodas @cryptoe I have submitted the first iteration of the patch to Ranger: https://issues.apache.org/jira/browse/RANGER-2395 . We are testing it in our production environment.

Couple of points:

Does not use private APIs and thus doesn't require any user synchronization. It does uses standard Hadoop user integration (local system (ipa, nss), LDAP etc).
Builds against Ranger master
No row masking etc yet as Presto lacks support
Supports Kerberos
Standard auditing, to be seen if this needs to be extended

sajjoseph · 2019-04-13T00:23:56Z

@cryptoe
Does this connector support Apache Ranger 1.2?

bolkedebruin · 2019-04-14T12:49:52Z

@Phelodas @cryptoe @dain the plugin is in review with Apache Ranger. I fyou have some time to spare and would like to ensure its quality please have a look at:

https://github.com/bolkedebruin/ranger/tree/presto_plugin from a Presto perspective you will be most interested in the shim: https://github.com/bolkedebruin/ranger/tree/presto_plugin/ranger-presto-plugin-shim
The full review is happing at https://reviews.apache.org/r/70394/
PR with Ranger is at [RANGER-2395] Add Presto plugin apache/ranger#36 , but will go through Jira most likely before being applied.

@sajjoseph the one that is in review with Apache Ranger (note: not the one in this PR) does support recent versions of Ranger. It is applied against master however.

sajjoseph · 2019-04-14T14:28:27Z

Thanks @bolkedebruin. I spent some time yesterday and got 244 working with ranger 1.2 version. I quickly checked the plugin added in ranger. I liked the brevity. I wonder how you are managing the dependency issue reported by @cryptoe in ranger plugin. Anyway, I had to dig a little with 244 PR to figure out how I can make everything work.

All the entries below might be obvious for others, but not for me.

If your hive source (presto catalog) name is different than the usual "hive", you will have to edit the ranger-security.xml and ranger-audit.xml files and add appropriate changes.

For example, in ranger-security.xml, you will find the following:

     <property>
         <name>ranger.plugin.hive.policy.cache.dir</name>
         <value>/etc/ranger/hive/policycache</value>
     </property>

It should be changed to:

    <property>
        <name>ranger.plugin.newhive.policy.cache.dir</name>
        <value>/etc/ranger/newhive/policycache</value>
    </property>

if the name of the catalog was newhive and similar changes through out the xml files.

User name in ranger is case sensitive. If presto's user used their id in lower case as an example, ranger will reject the request as the names won't match. Looks like there is a workaround for it in ranger - https://community.hortonworks.com/articles/145832/ranger-user-sync-issues-due-to-case-difference.html
I had to disable shading of "org.apache.hadoop". There were multiple packages looking for org.apache.hadoop... based classes and they started throw exception (Class not found).

If there is interest, I can get a PR for ranger 1.2 added here.

Next step is to incorporate kerberos based authentication in the plugin (initial authentication into ranger at the time of loading the plugin. I know it supports it, but not documented at this time).

Hope it helps.

bolkedebruin · 2019-04-14T15:08:29Z

The dependency issue is solved with a shim. That’s why is is now based on the source tree of Ranger rather than Presto as Ranger has the necessary infrastructure for this.

Furthermore, I don’t it’s a great idea to use the hive definitions for Presto and we made presto use its own, this also helps when the row filtering and column masking is used as you can than use ansi sql and no translation is required. The plugin in this Pr is also using private apis from Ranger for syncing users, which is not required. The plugin that is with Apache Ranger supports the latest release of presto.

In other words I don’t think the best place for the plugin is here (presto) but rather with Ranger.

Edit: both versions of the plugin already support Kerberos.

bolkedebruin · 2019-05-22T11:51:28Z

Support for Presto has been merged into Ranger (no row level security yet, as Presto lacks support at the moment).

apache/ranger@43757e7

sajjoseph · 2019-11-10T20:01:29Z

@cryptoe - I seem to have difficulty in getting the tag based policies working. For example, if we tag a certain table column and if there is a tag based policies in place, I expected the table suddenly come under the scrutiny of that policy and the defined access control checks take place.

If you can help me understand how the tag based policies will work with the current implementation, that will be great.

cryptoe · 2019-12-01T15:38:26Z

@sajjoseph https://github.com/prestosql/presto/pull/244/files?file-filters%5B%5D=.Plugin&file-filters%5B%5D=.java&file-filters%5B%5D=.json&file-filters%5B%5D=.txt&file-filters%5B%5D=.yaml#diff-2eaf70a9fd3dd682882245cec11822baR215

The test case should be a good starting point

jiegzhan · 2020-04-06T21:53:17Z

You guys have any timelines to merge this PR? Or is this abandoned?

bolkedebruin · 2020-04-07T00:44:19Z

@jkegzhan it is part of Ranger 2.0. Presto > 321 require a newer version that hasnt been merged yet.

In other words it has been working for around a year already.

bolkedebruin · 2020-04-07T00:44:36Z

This issue should be closed

tooptoop4 · 2020-05-04T21:07:28Z

@cryptoe any idea on https://jira.apache.org/jira/browse/RANGER-2816 ?

cryptoe added 3 commits February 15, 2019 22:11

WIP: Ranger integration of prestosql

f501ab6

Merge branch 'master' of https://github.com/prestosql/presto

30f7b1a

Version bump

0e01d41

dain changed the title ~~Wip Presto ranger integration~~ [WIP] Presto ranger integration Feb 20, 2019

cryptoe added 2 commits March 5, 2019 20:29

Adding ranger product tests

481fd13

1. Adding product test cases. 2. Merge things

168a878

cla-bot bot added the cla-signed label Mar 6, 2019

shubhamtagra reviewed Mar 18, 2019

View reviewed changes

cryptoe added 2 commits March 27, 2019 20:57

Making rangerSystemAccessControl work with the pluginClassloader

e7eb83b

Removing rowBaseFiltering

c5f7029

kokosing added WIP and removed WIP labels May 8, 2019

electrum closed this Apr 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Presto ranger integration #244

[WIP] Presto ranger integration #244

cryptoe commented Feb 15, 2019 •

edited

cla-bot bot commented Feb 15, 2019

shubhamtagra commented Feb 18, 2019

cryptoe commented Mar 6, 2019

shubhamtagra Mar 18, 2019

shubhamtagra Mar 18, 2019

cryptoe Mar 27, 2019

shubhamtagra Mar 18, 2019

cryptoe Mar 27, 2019

shubhamtagra commented Mar 18, 2019

bolkedebruin commented Mar 27, 2019

bolkedebruin commented Apr 4, 2019

sajjoseph commented Apr 13, 2019

bolkedebruin commented Apr 14, 2019 •

edited

sajjoseph commented Apr 14, 2019

bolkedebruin commented Apr 14, 2019 •

edited

bolkedebruin commented May 22, 2019

sajjoseph commented Nov 10, 2019

cryptoe commented Dec 1, 2019 •

edited

jiegzhan commented Apr 6, 2020

bolkedebruin commented Apr 7, 2020

bolkedebruin commented Apr 7, 2020

tooptoop4 commented May 4, 2020

[WIP] Presto ranger integration #244

[WIP] Presto ranger integration #244

Conversation

cryptoe commented Feb 15, 2019 • edited

cla-bot bot commented Feb 15, 2019

shubhamtagra commented Feb 18, 2019

cryptoe commented Mar 6, 2019

shubhamtagra Mar 18, 2019

Choose a reason for hiding this comment

shubhamtagra Mar 18, 2019

Choose a reason for hiding this comment

cryptoe Mar 27, 2019

Choose a reason for hiding this comment

shubhamtagra Mar 18, 2019

Choose a reason for hiding this comment

cryptoe Mar 27, 2019

Choose a reason for hiding this comment

shubhamtagra commented Mar 18, 2019

bolkedebruin commented Mar 27, 2019

bolkedebruin commented Apr 4, 2019

sajjoseph commented Apr 13, 2019

bolkedebruin commented Apr 14, 2019 • edited

sajjoseph commented Apr 14, 2019

bolkedebruin commented Apr 14, 2019 • edited

bolkedebruin commented May 22, 2019

sajjoseph commented Nov 10, 2019

cryptoe commented Dec 1, 2019 • edited

jiegzhan commented Apr 6, 2020

bolkedebruin commented Apr 7, 2020

bolkedebruin commented Apr 7, 2020

tooptoop4 commented May 4, 2020

cryptoe commented Feb 15, 2019 •

edited

bolkedebruin commented Apr 14, 2019 •

edited

bolkedebruin commented Apr 14, 2019 •

edited

cryptoe commented Dec 1, 2019 •

edited