Ozone+Ranger

[zorel]

Hi all.

I'm trying, for a POC, to configure Ozone and Ranger, manually fromn the binaries. I'm hitting an issue configuring the Ozone service in Ranger.

I had to manually copy common-lang3 and common-compress to make the enable-ozone-plugin script, which first seems strange, and in Ranger I have java.lang.NoClassDefFoundError: org/apache/hadoop/hdds/conf/ConfigurationSource. org.apache.hadoop.hdds.conf.ConfigurationSource. when i test the connection.

Do anyone succeed to make it work without Cloudera tools?

Any suggestion?

[kerneltime]

@zorel Are you tying to use Ranger for Ozone auth or trying to store Ranger logs in Ozone?

[yazynin]

You can show your ranger-ozone-audit.xml?

[zorel]

I don't have any ranger-ozone-audit.xml on the Ranger server. Those files belong belong to the plugin to be deployed on the Ozone side if I'm not wrong.

[yazynin]

it should be in /opt/ozone/ozone-1.4.0/etc/hadoop/ where Ozone Manager running

[yazynin]

And about your first question, ranger works with Ozone without Cloudera tools

[zorel]

I have issues to create the service in Ranger, I'm not yet on the Ozone configuration part.

[netscrol2016]

Hello @zorel , the process of setting up ozone + ranger has many pitfalls. This process took us about 5 days to figure out how the ranger works with ozone. I'll give you a hint, but would you like to know are you using Kerberos or simple password authentication?

[zorel]

om/< host >@BIGTOPPOC

[yazynin]

Try as is
<property><name>ozone.om.kerberos.principal</name><value>OM/_HOST@BIGTOPPOC</value></property>

[zorel]

well, my principal is in lowercase, not upper case. And Ozone works, the issue I have is with Ranger.

[zorel]

My bad in ozone-site, ozone.om.kerberos.principal is at om/_HOST@BIGTOPPOC.

[zorel]

Strange enough, I have to kinit before launching ranger otherwise UGI sees a SIMPLE authentication. With kinit to ranger admin before, I seems I can get Kerberos to work, with the issues before.

[zorel]

Gave up. Can't manage it to work. Neither with 2.4.0 or 3.0.0 snapshot.

[yazynin]

You're doing something wrong. Ozone in HA + kerberos + Ranger 2.4.0 works for us.

[zorel]

I'm sure I'm doing something wrong, that's why I'd like to know what steps you did to make it work.

[kerneltime]

@zorel maybe you can jump on one of the community syncs? The Asia time zone might work best for you.
11:15 PM PST on Thursdays: https://cloudera.zoom.us/j/95637039175

[zorel]

I installed CDP Trial to verify the parameters. Without kerberos, on a single node on GCP. With Ranger and Ozone. Out of the box: when I test the Ozone connection
"java.lang.NoClassDefFoundError: org/apache/hadoop/hdds/conf/ConfigurationSource.
org.apache.hadoop.hdds.conf.ConfigurationSource."

That's reassuring and depressing in the same time.

[kerneltime]

@zorel did you use Cloudera Manager? Cloudera Manager adds the ranger plugin jars to the Ozone class path. We use CDP cluster daily with Ranger integration. Let us know we can try to help you during one of the community syncs.

[zorel]

I installed the trial, and created the cluster through CM indeed. did not modify anything, and just selected Ozone and Ranger as component. I did not even try to use Ozone, the error comes from Ranger UI when I click on "test connection" on the cm_ozone component.
On my POC env, Ozone 1.4/Ranger 2.4/MIT kerberos, I still get this message, after a clean ranger installation and some modifications of jars, env:

"Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: 
Server has invalid Kerberos principal: om/<host>@MYREALM.COM, expecting: OM/<host>@REALM, while invoking $Proxy105.submitRequest over nodeId=null,nodeAddress=0.0.0.0"

It manage to read the good principal from the serveur (om/host@MYREALM.COM) but I can't understant why it expects OM/host@REALM (which is the default principal from

ozone/hadoop-hdds/common/src/main/resources/ozone-default.xml

Line 2230 in 9a6ece2

<value>OM/_HOST@REALM</value>

).

I will attend the next community sync next monday, works better for me (CET).

[kerneltime]

@zorel I don't think it is very helpful of me to say it works for the cluster I am using but for the case where it is a CDP distro (which version?) it should be fairly straightforward.. for the second issue we can look into kerberos config as part of the community sync.

[zorel]

Ok, writing this gave me an idea. I've already put some ozone parameter in the core-site.xml of ozone conf and set the conf/ranger-admin-env-hadoopconfdir.sh to something relevant, and add HADOOP_DIR to this be relevant for this file. So I just added the ozone.om.kerberos.principal to core-site.xml, and now it works.

The test connection works and I can autocomplete the volumes in the policy editor.

I had to do some more tricks to the installation, like changing the ranger.service.host of making a bunch of symlinks:

# ll
total 36
lrwxrwxrwx 1 root   root      62 Mar  5 16:03 bcpkix-jdk15on-1.67.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/bcpkix-jdk15on-1.67.jar
lrwxrwxrwx 1 root   root      62 Mar  5 22:51 bcprov-jdk15on-1.67.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/bcprov-jdk15on-1.67.jar
lrwxrwxrwx 1 root   root      59 Mar  5 16:03 guava-32.0.0-jre.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/guava-32.0.0-jre.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:03 hdds-client-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/hdds-client-1.4.0.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:03 hdds-common-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/hdds-common-1.4.0.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:03 hdds-config-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/hdds-config-1.4.0.jar
lrwxrwxrwx 1 root   root      70 Mar  5 16:03 hdds-interface-client-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/hdds-interface-client-1.4.0.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:01 jaeger-core-1.6.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/jaeger-core-1.6.0.jar
lrwxrwxrwx 1 root   root      65 Mar  5 16:01 opentracing-api-0.33.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/opentracing-api-0.33.0.jar
lrwxrwxrwx 1 root   root      66 Mar  5 16:01 opentracing-noop-0.33.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/opentracing-noop-0.33.0.jar
lrwxrwxrwx 1 root   root      75 Mar  5 16:01 opentracing-tracerresolver-0.1.8.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/opentracing-tracerresolver-0.1.8.jar
lrwxrwxrwx 1 root   root      66 Mar  5 16:01 opentracing-util-0.33.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/opentracing-util-0.33.0.jar
lrwxrwxrwx 1 root   root      61 Mar  5 16:03 ozone-client-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ozone-client-1.4.0.jar
lrwxrwxrwx 1 root   root      61 Mar  5 16:04 ozone-common-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ozone-common-1.4.0.jar
lrwxrwxrwx 1 root   root      71 Mar  5 16:05 ozone-interface-client-1.4.0.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ozone-interface-client-1.4.0.jar
lrwxrwxrwx 1 root   root      63 Mar  5 16:04 protobuf-java-3.19.6.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/protobuf-java-3.19.6.jar
-rw-r--r-- 1 ranger ranger 34323 Mar 31  2023 ranger-ozone-plugin-2.4.0.jar
lrwxrwxrwx 1 root   root      61 Mar  5 16:01 ratis-client-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-client-3.0.1.jar
lrwxrwxrwx 1 root   root      61 Mar  5 16:01 ratis-common-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-common-3.0.1.jar
lrwxrwxrwx 1 root   root      59 Mar  5 16:01 ratis-grpc-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-grpc-3.0.1.jar
lrwxrwxrwx 1 root   root      66 Mar  5 16:01 ratis-metrics-api-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-metrics-api-3.0.1.jar
lrwxrwxrwx 1 root   root      74 Mar  5 16:01 ratis-metrics-dropwizard3-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-metrics-dropwizard3-3.0.1.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:01 ratis-netty-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-netty-3.0.1.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:01 ratis-proto-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-proto-3.0.1.jar
lrwxrwxrwx 1 root   root      61 Mar  5 16:01 ratis-server-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-server-3.0.1.jar
lrwxrwxrwx 1 root   root      65 Mar  5 16:01 ratis-server-api-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-server-api-3.0.1.jar
lrwxrwxrwx 1 root   root      70 Mar  5 16:01 ratis-thirdparty-misc-1.0.5.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-thirdparty-misc-1.0.5.jar
lrwxrwxrwx 1 root   root      60 Mar  5 16:01 ratis-tools-3.0.1.jar -> /opt/ozone/ozone-1.4.0/share/ozone/lib/ratis-tools-3.0.1.jar
# pwd
/opt/ranger/ranger-2.4.0-admin/ews/webapp/WEB-INF/classes/ranger-plugins/ozone