apache/ozone:2.2.1 image cannot load libozone_rocksdb_tools.so

[greenwich]

Hi all,

After we upgraded from 2.0.0 to 2.2.1, every OM logs this at startup:

WARN  utils.NativeLibraryLoader (NativeLibraryLoader.java:loadLibrary(138)) - Unable to load library: ozone_rocksdb_tools
java.lang.UnsatisfiedLinkError: /opt/hadoop/ozone_rocksdb_tools<N>/libozone_rocksdb_tools.so:
  /lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by .../libozone_rocksdb_tools.so)
    at org.apache.hadoop.hdds.utils.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:133)
    at org.apache.hadoop.hdds.utils.db.ManagedRawSSTFileReader.loadLibrary(ManagedRawSSTFileReader.java:55)
    at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.initNativeLibraryForEfficientDiff(SnapshotDiffManager.java:306)
    ...

It reproduces with the stock apache/ozone:2.2.1 image from Docker Hub, without any of our changes:

docker run --rm --platform linux/amd64 --entrypoint sh apache/ozone:2.2.1 -c '
  grep -E "^(NAME|VERSION)=" /etc/os-release
  strings /lib64/libstdc++.so.6 | grep -o "GLIBCXX_3\.4\.[0-9]*" | sort -uV | tail -1
  cd /tmp && jar xf /opt/hadoop/share/ozone/lib/hdds-rocks-native-2.2.1.jar
  strings libozone_rocksdb_tools.so | grep -m1 "GCC: ("
  strings libozone_rocksdb_tools.so | grep -o "GLIBCXX_3\.4\.[0-9]*" | sort -uV | tail -1'

Output:

NAME="Rocky Linux"
VERSION="9.8 (Blue Onyx)"
GLIBCXX_3.4.29                    <- what the runtime provides
GCC: (Debian 14.2.0-19) 14.2.0    <- how the .so was built
GLIBCXX_3.4.32                    <- what the .so needs

It looks like the native library in the 2.2.1 release was built on a Debian 13 / GCC 14 host. It links libstdc++ dynamically, and there is no -static-libstdc++ in hadoop-hdds/rocks-native. The image runtime, though, is apache/ozone-runner on Rocky 9, whose libstdc++ is GCC 11. So the official image cannot load its own native library. SnapshotDiffManager falls back to a full diff, and RocksDBCheckpointDiffer skips SST pruning (Cannot prune OMKeyInfo from SST files).

We don't use snapshots, so for us it's just log noise; we set ozone.om.snapshot.load.native.lib=false. Anyone who does use snapshot diff on the official image is silently getting the slow path, though.

Questions:

1. Is this known or tracked? I couldn't find an HDDS issue mentioning GLIBCXX or this error.
2. Was 2.0.0 or 2.1.x built differently, i.e. is this a regression of the 2.2.1 release build?
3. What is the preferred fix? Some options:
   • build rocks-native for releases on the runner's OS (Rocky 9 + gcc-toolset), or inside ozone-runner;
   • link the library with -static-libstdc++;
   • add a release check that runs ozone debug checknative inside the published image, so a mismatch fails the release instead of only logging a WARN.
4. Since HDDS-14226 plans to remove this native library in favour of the RocksDB JNI API, is it worth fixing the build for 2.2.x, or is the advice to disable the native lib until then?

I'm happy to open a JIRA and help with a patch if it's useful.