Current quay.io/openebs/zfs-driver:ci image is not provisioning on CentOS 8

[jlcox1970]

What steps did you take and what happened:
cleaned up all openebs config in cluster, remove all zfs based pvc and reinstalled using the README

What did you expect to happen:
expected to be able to run 'zfs' inside openebs-zfs-plugin container as per previous install

The output of the following commands will help us better understand what's going on:

zfs: error while loading shared libraries: libtirpc.so.3: cannot open shared object file: No such file or directory

Anything else you would like to add:
I have tried with the yaml from "Support for deploying to CentOS #119" but this still failed due to changing libs in Centos8

Environment:

• ZFS-LocalPV version
  v0.9.2

• Kubernetes version (use kubectl version):
  Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
  as above

• Cloud provider or hardware configuration:

bear metal,
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node-01 Ready 104d v1.18.3 10.1.1.221 CentOS Linux 8 (Core) 4.18.0-147.8.1.el8_1.x86_64 docker://19.3.12

vm-host.globelock.home Ready master 104d v1.18.3 10.1.1.220 CentOS Linux 8 (Core) 4.18.0-193.14.2.el8_2.x86_64 docker://19.3.12

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="8 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="8"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="CentOS Linux 8 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:8"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

[pawanpraka1]

@jlcox1970 which operator yaml you used to deploy? For centos, the operator yaml is different

CentOS 7:
https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/operators/centos7/zfs-operator.yaml

CentOS 8:
https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/operators/centos8/zfs-operator.yaml

[jlcox1970]

both the default fro theREADME and the CentOS 8 operator
the CentOs8 gives
zfs: symbol lookup error: /lib/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b

ldd /usr/sbin/zfs
linux-vdso.so.1 (0x00007ffd45bf5000)
libnvpair.so.1 => /lib/libnvpair.so.1 (0x00007efec5532000)
libuutil.so.1 => /lib/libuutil.so.1 (0x00007efec5323000)
libzfs.so.2 => /lib/libzfs.so.2 (0x00007efec50a7000)
libzfs_core.so.1 => /lib/libzfs_core.so.1 (0x00007efec4ea1000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007efec4e4a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efec4cfb000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007efec4c67000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007efec4992000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efec4987000)
libtirpc.so.3 => /lib/libtirpc.so.3 (0x00007efec4754000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007efec4729000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007efec4720000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efec4702000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efec46df000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efec44ee000)
/lib64/ld-linux-x86-64.so.2 (0x00007efec5970000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007efec44d4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efec44ce000)
libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x00007efec427c000)
libkrb5.so.3 => /lib/libkrb5.so.3 (0x00007efec3f8c000)
libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x00007efec3d70000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007efec3d69000)
libkrb5support.so.0 => /lib/libkrb5support.so.0 (0x00007efec3b58000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00007efec3954000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007efec3937000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007efec390c000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007efec3887000)

the paths for libssl and libcrypto have changed and this is from inside the pod

ls -l /usr/lib/libcrypto.so.10 /usr/lib/libssl.so.10
-rw-r--r--. 1 root root 0 Jun 8 23:04 /usr/lib/libcrypto.so.10
-rw-r--r--. 1 root root 0 Jun 8 23:04 /usr/lib/libssl.so.10

so it looks like we need to update the volume mounts again

[pawanpraka1]

Playing with the library mount is causing a lot of trouble. Let me take this up in this release which is planned on 15th of Sept. I will do the root mount which will fix it for all the operating systems.

[jlcox1970]

available for pre release testing if needed

[pawanpraka1]

Thanks @jlcox1970, will ping you to test the changes.

[pawanpraka1]

@jlcox1970 I have pushed the changes to my repo and I have verified it on ubuntu. Will do the test on centos 7 and 8 tomorrow. Here is the yaml : https://raw.githubusercontent.com/pawanpraka1/zfs-localpv/rootmnt/deploy/zfs-operator.yaml.

it is still work in progress, let me know if this yaml is working for you. Note that this yaml has IfNotPresent imagepull policy, so if you want you can use the zfs-driver:0.9.2 release.

PR: #204

[jlcox1970]

tested on CentOS8
now works as expected and I have provisioned pvc's using the provisioner

unable to test on CentOS7 as I do not have access

[pawanpraka1]

Thanks for verifying this @jlcox1970. I hope now this fixes for all the operating systems and we don't have to worry about the libraries on different platforms.