Ozone HA secure v1.3.0 doesn't work from released artifact

[MerelyOneMax1986]

Hi community,
I've tried the guide from https://github.com/apache/ozone#run-ozone-from-released-artifact , in particular, I've used 1.3.0 version of
ozonesecure-ha in compose folder.
Once cluster up, I've ended up failing to put a simple file to Ozone HA secure via a standard hdfs tool:

hdfs dfs -put /etc/hosts /volume1/bucket1/test

Here you are the logs:
put_file_to_ozone_ha.log

I've picked example as is, so could you please help me to fix that? Thanks in advance!

[errose28]

Hi @MerelyOneMax1986, based on the client log you attached it looks like the client is unable to communicate with the datanodes to write a Ratis-replicated object. Could you please attach logs for the 3 datanodes? In the default docker setup these are usually configured to go to docker compose logs <container> instead of a file in the container.

[MerelyOneMax1986]

Hi @errose28, thanks a lot for your response to my problem.

Here you are the datanodes' logs:
datanode3.log
datanode2.log
datanode1.log

Also, I wanted to share my core-site.xml to connect the cluster:

<configuration>
<property>
  <name>fs.ofs.impl</name>
  <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
</property>
<property>
  <name>fs.defaultFS</name>
  <value>ofs://id1</value>
</property>
<property>
  <name>ozone.om.service.ids</name>
  <value>id1</value>
</property>
<property>
  <name>ozone.om.nodes.id1</name>
  <value>om1,om2,om3</value>
</property>
<property>
  <name>ozone.om.address.id1.om1</name>
  <value>om1</value>
</property>
<property>
  <name>ozone.om.address.id1.om2</name>
  <value>om2</value>
</property>
<property>
  <name>ozone.om.address.id1.om3</name>
  <value>om3</value>
</property>
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>
<property>
  <name>ozone.om.kerberos.principal</name>
  <value>om/om@EXAMPLE.COM</value>
</property>
</configuration>

hdfs-site.xml:

<configuration>
<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>
<property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:1019</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:1012</value>
</property>
</configuration>

[MerelyOneMax1986]

I believe OM should know how to connect to Datanodes as well as needed ports. The addresses with ports for Datanodes I took from the example docker-config.

So, I've been trying to play with datanodes' ports or even delete the lines, still no success.

[errose28]

The datanode logs look normal. Only thing I see, which is unrelated to this issue, is the volume scanner running way too frequently. Did you make a config change to any of the hdds.datanode.disk.check* configs? I haven't seen this on any other cluster using default configs here.

2023-07-26 07:15:06,278 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/metadata/ratis
2023-07-26 07:15:06,278 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/metadata/ratis
2023-07-26 08:15:06,279 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
2023-07-26 08:15:06,280 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
2023-07-26 08:15:06,280 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/metadata/ratis
2023-07-26 08:15:06,280 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/metadata/ratis
2023-07-26 09:15:06,280 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
2023-07-26 09:15:06,281 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
2023-07-26 09:15:06,281 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/metadata/ratis
2023-07-26 09:15:06,281 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/metadata/ratis

[errose28]

Ozone does not use dfs.datanode.address or dfs.datanode.http.address AFAIK. I think they are just in there for some legacy reason. Other docker compose configs do not have them defined. OM does not communicate directly with datanodes in Ozone, so it should not read any datanode related configs. Datanodes will register to SCM, and SCM will use this information to provide data location information to OM.

[MerelyOneMax1986]

From my point of view, in any core-site.xml configs like it was in HDFS we mention namenodes, in HDFS namenodes match Ozone managers (om). In HDFS, namenodes give what and where data chunks are resided, then clients go to datanodes directly. I believe as we talk only about om in client configs, then they act like namenodes plus/minus and give information about datanodes. Due to we don't mention any datanodes addresses as well as scm.

Highly likely, om goes to scm and give datanodes and data location as well to clients.

[errose28]

I re-tested the ha-secure docker compose setup from 1.3.0 (CI would have run this anyways) and it works for me. The only thing I think I am doing differently is I am using the ozone client from within the already defined containers. For example ozone fs -put ... from within the om1 container. There is no HDFS client in the containers by default like the one being used for testing, which brings me to my next questions:

• Where is your HDFS client located?
  • Did you add it to the containers you are trying to use?
  • Is it being used from outside docker to query docker using port forwarding?
  • Is it in a new container added to the cluster?
• Does it work if you run commands from within one of the already defined containers?
• Do any of the simpler, non-secure clusters work for you, like the one in compose/ozone?
• Are you testing on an x86 machine? 1.3.0 docker image was only built for that architecture so if running it on arm (like apple silicon) you may get issues due to how slow it will run.
• A quick check that this cluster was not started with --scale datanode=3. That would be the way to get 3 datanodes in the compose/ozone cluster used in the quick start, but secure-ha has one service defined per datanode, so --scale datanode1=3 or may cause issues.

[MerelyOneMax1986]

@errose28 a good piece of advice, so just to reproduce, you are able to add the following lines to docker-compose.yaml:

  hdfstool:
    image: docker-registry.dp.nlmk.com/buslovaev/hdfstool
    hostname: hdfstool
    command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]
    volumes:
      - ./core-site.xml:/HADOOP_CONF_DIR/core-site.xml
      - ./hdfs-site.xml:/HADOOP_CONF_DIR/hdfs-site.xml
      - ./krb5.conf:/etc/krb5.conf
      - ../_keytabs:/etc/security/keytabs
    networks:
      ozone_net:
        ipv4_address: 172.25.0.254

Then you'll do kinit, and then try to put a simple file via hdfs dfs -put

BTW, the cluster was started with docker-compose up -d as in docs.

Any other options in compse/ozone I didn't try as it's not connected to my task.

Interesting, I've tried your idea about ozone fs and I am surprised of the results:

bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
bash-4.2$ ozone fs -mkdir /volume1
bash-4.2$ ozone fs -mkdir /volume1/bucket1
bash-4.2$ ozone fs -put /etc/hosts /volume1/bucket1/test
bash-4.2$ ozone fs -ls /volume1/bucket1/test
-rw-rw-rw-   1 testuser testuser        236 2023-07-28 15:00 /volume1/bucket1/test

Could you please advise me what is wrong with hdfstool?
So, my final goal is Spark client successful connection to Ozone HA via docker-compose.
And the error is the same when running the hdfstool command. According to the docs https://ozone.apache.org/docs/current/interface/ofs.html it should work.
Thanks in advance!

P.S:
I've noticed that apache/ozone-runner:20220623-1 is different than apache/ozone:1.3.0 (kubernetes example)
Do you know the full answer? AFAIK, runner tags are just for testing purposes, not for production.