RANGER-4640: Trino ranger plugin for 433 snapshot #291

shreyas-dview · 2023-12-18T06:51:09Z

What changes were proposed in this pull request?

Changes made to support Trino Ranger Plugin with 433 version. Also made ranger compatible by adding nashorn-core to agent-commons to support build it with java 11 and java 17.

How was this patch tested?

Yes this is been deployed with our prod env and is working fine for more than an month now.

…-plugin

origin0099 · 2024-01-12T02:31:02Z

Your efforts have been of great help to me. thank you

lozbrown · 2024-01-15T13:21:53Z

is this likely to get reviewed/merged soon?

mneethiraj · 2024-01-17T20:53:51Z

@shreyas-dview - this PR includes binary file ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz. Can you please remove this file and update the PR?

Would 17 be the minimum JDK version after this patch? Given many existing plugins require JDK8 support, it is critical to be able to build with earlier version JDK as well.

… trino-ranger-plugin

shreyas-dview · 2024-01-18T09:46:50Z

@mneethiraj removed ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz, thanks for pointing that out.

Would 17 be the minimum JDK version after this patch? -> if you look at pom.xml changes I have removed trino-plugin from default and jdk-11 profile , so if anyone wants to build trino-plugin they have to use -Pranger-jdk17 profile

mneethiraj · 2024-01-18T21:16:23Z

if you look at pom.xml changes I have removed trino-plugin from default and jdk-11 profile , so if anyone wants to build trino-plugin they have to use -Pranger-jdk17 profile

@shreyas-dview - build fails with the following error due to changes to agents-common module. Can you please review and update?

[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ ranger-plugins-common ---
[INFO] Compiling 250 source files to /home/build/Apache/git/ranger/agents-common/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/build/Apache/git/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[26,40] error: cannot access ClassFilter
[INFO] 1 error
[INFO] -------------------------------------------------------------

bhavikpatel9977 · 2024-01-22T03:44:58Z

Nashron is not available from jdk15+.
You can exclude from the build and use graalvm for jdk-17.

The dependency is already added in the plugins-common module, you can enable that dependency and try to build.

origin0099 · 2024-01-22T07:34:31Z

ZooKeeper 3.5 has a transitive dependency on a zookeeper-jute artifact.
./distro/src/main/assembly/plugin-trino.xml should have
"org.apache.zookeeper:zookeeper-jute:jar:${zookeeper.version}"

The latest version of Trino supports the update statement.
vim ./ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java

246 //kimdy
247 @OverRide
248 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames) {
249 try {
250 activatePluginClassLoader();
251 systemAccessControlImpl.checkCanUpdateTableColumns(securityContext, table, updatedColumnNames);
252 } finally {
253 deactivatePluginClassLoader();
254 }
255 }

vim ./plugin-trino/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java

631 // kimdy
632 @OverRide
633 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames){
634 for (RangerTrinoResource res : createResource(table, updatedColumnNames)) {
635 if (!hasPermission(res, securityContext, TrinoAccessType.UPDATE)) {
636 LOG.debug("RangerSystemAccessControl.checkCanUpdateTableColumns(" + table.getSchemaTableName().getTableName() + ") denied");
637 AccessDeniedException.denyUpdateTableColumns(table.getSchemaTableName().getTableName(), updatedColumnNames);
638 }
639 }
640 }

// UDPATE 추가
923 enum TrinoAccessType {
924 CREATE, DROP, SELECT, INSERT, DELETE, USE, ALTER, ALL, GRANT, REVOKE, SHOW, IMPERSONATE, EXECUTE, UPDATE;
925 }

나의 빌드 환경
java -version
openjdk version "17.0.7" 2023-04-18 LTS
OpenJDK Runtime Environment Zulu17.42+19-CA (build 17.0.7+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.42+19-CA (build 17.0.7+7-LTS, mixed mode, sharing)

mvn -v
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /opt/apps/maven
Java version: 17.0.7, vendor: Azul Systems, Inc., runtime: /opt/apps/zulu17.42.19-ca-jdk17.0.7-linux_x64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"

mvn clean package -DskipTests -P ranger-trino-plugin,-linux -am -pl distro,plugin-trino,ranger-trino-plugin-shim,agents-installer,credentialbuilder

Lastly, ranger's group policy doesn't seem to apply well to trino. The role policy applies well to trino.
I'm an old school poor IT guy who's not familiar with GIT, so I don't know what form and procedure I should use to submit a proposal. I hope this was of some help. Your efforts have been of great help to me. thank you!

respon007 · 2024-01-23T11:31:24Z

@shreyas-dview - this PR includes binary file ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz. Can you please remove this file and update the PR?

Would 17 be the minimum JDK version after this patch? Given many existing plugins require JDK8 support, it is critical to be able to build with earlier version JDK as well.

I had resolved the problem by maven-tool-chains plugin (https://stackoverflow.com/questions/12414209/compile-maven-module-with-different-java-version), but my ranger version is 2.3.

shreyas-dview · 2024-01-25T06:34:13Z

Give me some time over the weekend, will try to resolve the issues (if exists) or will answer them with proper explanation.

respon007 · 2024-01-25T13:27:04Z

ZooKeeper 3.5 has a transitive dependency on a zookeeper-jute artifact. ./distro/src/main/assembly/plugin-trino.xml should have "org.apache.zookeeper:zookeeper-jute:jar:${zookeeper.version}"

The latest version of Trino supports the update statement. vim ./ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java

246 //kimdy 247 @OverRide 248 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames) { 249 try { 250 activatePluginClassLoader(); 251 systemAccessControlImpl.checkCanUpdateTableColumns(securityContext, table, updatedColumnNames); 252 } finally { 253 deactivatePluginClassLoader(); 254 } 255 }

vim ./plugin-trino/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java

631 // kimdy 632 @OverRide 633 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames){ 634 for (RangerTrinoResource res : createResource(table, updatedColumnNames)) { 635 if (!hasPermission(res, securityContext, TrinoAccessType.UPDATE)) { 636 LOG.debug("RangerSystemAccessControl.checkCanUpdateTableColumns(" + table.getSchemaTableName().getTableName() + ") denied"); 637 AccessDeniedException.denyUpdateTableColumns(table.getSchemaTableName().getTableName(), updatedColumnNames); 638 } 639 } 640 }

// UDPATE 추가 923 enum TrinoAccessType { 924 CREATE, DROP, SELECT, INSERT, DELETE, USE, ALTER, ALL, GRANT, REVOKE, SHOW, IMPERSONATE, EXECUTE, UPDATE; 925 }

나의 빌드 환경
java -version
openjdk version "17.0.7" 2023-04-18 LTS
OpenJDK Runtime Environment Zulu17.42+19-CA (build 17.0.7+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.42+19-CA (build 17.0.7+7-LTS, mixed mode, sharing)

mvn -v Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39) Maven home: /opt/apps/maven Java version: 17.0.7, vendor: Azul Systems, Inc., runtime: /opt/apps/zulu17.42.19-ca-jdk17.0.7-linux_x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"

mvn clean package -DskipTests -P ranger-trino-plugin,-linux -am -pl distro,plugin-trino,ranger-trino-plugin-shim,agents-installer,credentialbuilder

Lastly, ranger's group policy doesn't seem to apply well to trino. The role policy applies well to trino.

I'm an old school poor IT guy who's not familiar with GIT, so I don't know what form and procedure I should use to submit a proposal. I hope this was of some help. Your efforts have been of great help to me. thank you!

@origin0099 I have tried your way,but the "Permissions" had no "Update" option. Could you help me?

origin0099 · 2024-01-29T05:00:08Z

I omitted the part where I had to manually modify RANGER's BACK-END DB. I'm using POSTGRESQL and used DBeaver.

I hope it helps you as it helped me.

respon007 · 2024-01-30T02:38:11Z

I omitted the part where I had to manually modify RANGER's BACK-END DB. I'm using POSTGRESQL and used DBeaver.

I hope it helps you as it helped me.

@origin0099 Thanks for your reply! I modify the file agents-common/src/main/resources/service-defs/ranger-servicedef-trino.json and re-compile ranger. Now I have solved the problem.

tarekabouzeid · 2024-01-30T13:57:03Z

Hi,

Thanks so much for this contribution, It helped me a lot. I have tested it with Trino 433 and Trino server is up and running and there seems to be an issue in Trino ranger raudits initialization when using Elastic search, so audits are not sent to ranger.

I am going to look more into this issue and will report my findings.

Elastic search version used : 7.16.2
Trino version: 433
Ranger: 2.4.0

Below is the error I am getting:

024-01-30T13:32:13.562Z    INFO    main    stdout  INFO  - Connecting to ElasticSearch: User:ranger, http://es-internal.infra:80/ranger_audits
2024-01-30T13:32:13.895Z    INFO    main    stdout  WARN  - request [POST http://es-internal.infra:80/ranger_audits/_open?master_timeout=30s&ignore_throttled=false&ignore_unavailable=false&expand_wildcards=closed&allow_no_indices=true&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.17.16-2b23fa076334f8d4651aeebe458a955a2ae23218 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2024-01-30T13:32:13.899Z    INFO    main    stdout  ERROR - Can't connect to ElasticSearch server: User:ranger, http://es-internal.infra:80/ranger_auditsjava.lang.BootstrapMethodError: bootstrap method initialization exception
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:188)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:312)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
    at org.elasticsearch.client.IndicesClient.open(IndicesClient.java:614)
    at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.newClient(ElasticSearchAuditDestination.java:267)
    at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.getClient(ElasticSearchAuditDestination.java:187)
    at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.init(ElasticSearchAuditDestination.java:101)
    at org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
    at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
    at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl.<init>(RangerSystemAccessControl.java:120)
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
    at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl.<init>(RangerSystemAccessControl.java:69)
    at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl$$FastClassByGuice$$1345336.GUICE$TRAMPOLINE(<generated>)
    at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl$$FastClassByGuice$$1345336.apply(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:33)
    at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
    at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
    at com.google.inject.Guice.createInjector(Guice.java:87)
    at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:262)
    at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53)
    at io.trino.spi.security.SystemAccessControlFactory.create(SystemAccessControlFactory.java:30)
    at io.trino.security.AccessControlManager.createSystemAccessControl(AccessControlManager.java:198)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.Collections$2.tryAdvance(Collections.java:5073)
    at java.base/java.util.Collections$2.forEachRemaining(Collections.java:5081)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at io.trino.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:167)
    at io.trino.server.Server.doStart(Server.java:162)
    at io.trino.server.Server.lambda$start$0(Server.java:91)
    at io.trino.$gen.Trino_433____20240130_133147_1.run(Unknown Source)
    at io.trino.server.Server.start(Server.java:91)
    at io.trino.server.TrinoServer.main(TrinoServer.java:38)
Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type interface org.apache.http.Header; not a subtype of implementation type interface org.apache.http.NameValuePair
    at java.base/java.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:273)
    at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:340)
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    ... 57 common frames omitted

nil1729 · 2024-02-03T18:26:05Z

Hello @tarekabouzeid, How did you build the ranger plugin, and have you setup TLS on elasticsearch ? I was getting connection error initially with ranger when there is TLS enabled on elasticsearch.
I have setup ranger with elasticsearch and trino 433, you can take a look once https://github.com/nil1729/trino-ranger-noob

tarekabouzeid · 2024-02-04T20:18:32Z

Hi @nil1729
Thank you for sharing, I guess its related to different JDK versions between Trino and Ranger, which version have you used to build ranger and which one you are using to run Trino ?

For me, I build ranger using : maven:3.8.5-openjdk-17-slim, it was built fine but ES was not working in Trino.
When I tried to build ranger using : maven:3.9.6-eclipse-temurin-17-focal, built failed

I got this error while building,

#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[53,36] error: cannot find symbol
#7 191.6 [ERROR]   symbol:   class TimeValue
#7 191.6 [ERROR]   location: package org.elasticsearch.common.unit
#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[54,40] error: cannot find symbol
#7 191.6 [ERROR]   symbol:   class XContentType
#7 191.6 [ERROR]   location: package org.elasticsearch.common.xcontent
#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[253,49] error: cannot find symbol

Trino is using Temurin JDK in its official docker image.

nil1729 · 2024-02-05T20:20:59Z

Hi @tarekabouzeid, I built the ranger locally with the JDK version openjdk-17.0.8 and maven version 3.6.3. There was some error when building on MAC due to the shadow.h header file not being found. Then I used those jars to build docker image and tested trino with docker only

shreyas-dview added 4 commits December 5, 2023 23:50

Created ranger plugin

3ad94e4

Fix distro issue

01b4ba6

Merge branch 'master' of github.com:dview-io/ranger into trino-ranger…

7f71e64

…-plugin

Comment trino module for java11 build

d2b4285

shreyas-dview changed the title ~~Trino ranger plugin for 433 snapshot~~ RANGER-4640 - Trino ranger plugin for 433 snapshot Jan 9, 2024

shreyas-dview changed the title ~~RANGER-4640 - Trino ranger plugin for 433 snapshot~~ RANGER-4640: Trino ranger plugin for 433 snapshot Jan 9, 2024

shreyas-dview added 4 commits January 18, 2024 15:06

Delete trino-plugin tar

1f6bf1f

Fix pom spacing issue

2c3c69e

Fix pom spacing issue

dcc49e7

Merge branch 'trino-ranger-plugin' of github.com:dview-io/ranger into…

63fc169

… trino-ranger-plugin

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RANGER-4640: Trino ranger plugin for 433 snapshot #291

RANGER-4640: Trino ranger plugin for 433 snapshot #291

shreyas-dview commented Dec 18, 2023 •

edited

origin0099 commented Jan 12, 2024

lozbrown commented Jan 15, 2024

mneethiraj commented Jan 17, 2024

shreyas-dview commented Jan 18, 2024

mneethiraj commented Jan 18, 2024

bhavikpatel9977 commented Jan 22, 2024

origin0099 commented Jan 22, 2024 •

edited

respon007 commented Jan 23, 2024

shreyas-dview commented Jan 25, 2024

respon007 commented Jan 25, 2024

origin0099 commented Jan 29, 2024 •

edited

respon007 commented Jan 30, 2024

tarekabouzeid commented Jan 30, 2024

nil1729 commented Feb 3, 2024

tarekabouzeid commented Feb 4, 2024 •

edited

nil1729 commented Feb 5, 2024 •

edited

RANGER-4640: Trino ranger plugin for 433 snapshot #291

Are you sure you want to change the base?

RANGER-4640: Trino ranger plugin for 433 snapshot #291

Conversation

shreyas-dview commented Dec 18, 2023 • edited

What changes were proposed in this pull request?

How was this patch tested?

origin0099 commented Jan 12, 2024

lozbrown commented Jan 15, 2024

mneethiraj commented Jan 17, 2024

shreyas-dview commented Jan 18, 2024

mneethiraj commented Jan 18, 2024

bhavikpatel9977 commented Jan 22, 2024

origin0099 commented Jan 22, 2024 • edited

respon007 commented Jan 23, 2024

shreyas-dview commented Jan 25, 2024

respon007 commented Jan 25, 2024

origin0099 commented Jan 29, 2024 • edited

respon007 commented Jan 30, 2024

tarekabouzeid commented Jan 30, 2024

nil1729 commented Feb 3, 2024

tarekabouzeid commented Feb 4, 2024 • edited

nil1729 commented Feb 5, 2024 • edited

shreyas-dview commented Dec 18, 2023 •

edited

origin0099 commented Jan 22, 2024 •

edited

origin0099 commented Jan 29, 2024 •

edited

tarekabouzeid commented Feb 4, 2024 •

edited

nil1729 commented Feb 5, 2024 •

edited